Solid-state imaging device, solid-state imaging method, and electronic apparatus

ABSTRACT

A solid-state imaging device includes a pixel array and a pixel value correcting unit. The pixel array includes a plurality of pixels, the plurality of pixels each having one of a different exposure time and a different exposure sensitivity and being disposed according to a predetermined rule. The pixel value correcting unit is configured to correct, among pixel values obtained from the plurality of pixels in the pixel array, a pixel value of a pixel of the plurality of pixels that applies to a preset condition, by using a pixel value of another pixel of the plurality of pixels.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2013−145913 filed Jul. 11, 2013, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present disclosure relates to a solid-state imaging device, asolid-state imaging method, and an electronic apparatus, and moreparticularly, to a solid-state imaging device, a solid-state imagingmethod, and an electronic apparatus that are capable of executingefficient defect correction processing in an image sensor capable ofcapturing an image in a high dynamic range.

From the past, pixel signals related to defective pixels have beencorrected in image signals output from an image sensor.

The defective pixels are normally classified into blown-out highlights(white-dot defective pixels) and blocked-up shadows (black-dot defectivepixels). The blown-out highlights (white-dot defective pixels)constantly output pixel signals having extremely large values. Theblocked-up shadows (black-dot defective pixels) constantly output pixelsignals having extremely small values. The correction processing of thepixel signals related to such defective pixels, that is, defectcorrection processing, is performed by calculating and generating thepixel signals of the defective pixels based on the pixel signals ofpixels located around the defective pixels.

Further, for example, in a pixel array of the Bayer array, a techniqueof selecting pixels, which are to be used for correction of defectivepixels, from pixels having the same color as the defective pixels isalso proposed (see, for example, Japanese Patent Application Laid-openNo. 2010−130238).

Moreover, in recent years, a method of enlarging the dynamic range of animage sensor has been proposed. For example, there has been developed atechnique of obtaining an image in a high dynamic range by changing anexposure time in accordance with the position of a pixel in the pixelarray of the Bayer array, the image in the high dynamic range beingappropriately displayed from the pixels of low luminance to the pixelsof high luminance. Such high-dynamic-range-image-capturing methodincludes an SVE (Spatially Varying Exposure) method and the like.

SUMMARY

In the correction of the defective pixels in related art, however, thedefect correction processing for the image sensor of the SVE method hasnot been taken into consideration.

The present disclosure has been made in view of such circumstances andit is desirable to allow an image sensor capable of capturing an imagein a high dynamic range to execute efficient defect correctionprocessing.

According to a first embodiment of the present disclosure, there isprovided a solid-state imaging device including a pixel array and apixel value correcting unit. The pixel array includes a plurality ofpixels, the plurality of pixels each having one of a different exposuretime and a different exposure sensitivity and being disposed accordingto a predetermined rule. The pixel value correcting unit is configuredto correct, among pixel values obtained from the plurality of pixels inthe pixel array, a pixel value of a pixel of the plurality of pixelsthat applies to a preset condition, by using a pixel value of anotherpixel of the plurality of pixels.

The pixel array may include the plurality of pixels each having one ofthe same exposure time and the same exposure sensitivity and beingregularly disposed on a row-by-row basis.

The pixel array may include the plurality of pixels including apredetermined number of pixels each having one of the same exposure timeand the same exposure sensitivity and being regularly disposed as anL-shaped group of pixels.

The pixel value correcting unit may be configured to set a pixel ofinterest in the plurality of pixels disposed in the pixel array to bethe center of the pixel array, extract a processing unit area includinga preset number of rows of pixels, and correct the pixel value of thepixel of interest for each processing unit area.

The processing unit area may include five rows.

The pixel value correcting unit may include a saturation determiningunit, a flatness determining unit, a direction detecting unit, a defectdetermining unit, and a defect correcting unit. The saturationdetermining unit may be configured to determine whether the processingunit area is saturated or not based on the number of pixels that outputa maximum pixel value among the pixels of the processing unit area. Theflatness determining unit may be configured to determine whether or notan image formed of the pixels of the processing unit area is a flatimage that is free from a texture. The direction detecting unit may beconfigured to detect a direction of the texture when it is determinedthat the image formed of the pixels of the processing unit area is not aflat image. The defect determining unit may be configured to determinewhether the pixel of interest is a defective pixel or not. The defectcorrecting unit may be configured to correct the pixel value of thepixel of interest when it is determined that the pixel of interest is adefective pixel.

In accordance with a result of the determination by the saturationdetermining unit, the flatness determining unit may be configured todetermine whether the image is a flat image or not, and the directiondetecting unit may be configured to detect the direction of the texture,by different methods.

In accordance with a result of the determination by the flatnessdetermining unit, the defect determining unit may be configured todetermine whether the pixel of interest is a defective pixel or not, andthe defect correcting unit may be configured to correct the pixel valueof the pixel of interest, by different methods.

The defect correcting unit may be configured to correct the pixel valueof the pixel of interest by replacing the pixel value of the pixel ofinterest with a pixel value of a pixel selected based on the detecteddirection of the texture, when it is determined that the image formed ofthe pixels of the processing unit area is not a flat image.

In the case where the pixel array includes the plurality of pixelsincluding a predetermined number of pixels each having one of the sameexposure time and the same exposure sensitivity and being regularlydisposed as an L-shaped group of pixels, and when the detected directionof the texture is a vertical direction, the defect correcting unit maybe configured to generate the pixel value of the pixel selected based onthe direction of the texture by linear interpolation.

The defect correcting unit may be configured to mix the pixel valuegenerated by the linear interpolation and the pixel value of the pixelof interest, based on a mixing ratio determined based on the pixel valuegenerated by the linear interpolation.

The solid-state imaging device may further include a gain adding unitconfigured to multiply, among the pixels of the processing unit area, apixel value of a pixel having one of a first exposure time and a firstexposure sensitivity by a predetermined gain, to thereby normalize thepixel values of the pixels of the processing unit area, with a pixelvalue of a pixel having one of a second exposure time and a secondexposure sensitivity being as a reference.

The solid-state imaging device may include a lamination-type imagesensor including a first chip on which the pixel array is disposed, anda second chip including a circuit for achieving a function of the pixelvalue correcting unit.

According to a second embodiment of the present disclosure, there isprovided a solid-state imaging method including correcting, among pixelvalues obtained from a plurality of pixels in a pixel array, a pixelvalue of a pixel of the plurality of pixels that applies to a presetcondition, by using a pixel value of another pixel of the plurality ofpixels, the plurality of pixels each having one of a different exposuretime and a different exposure sensitivity and being disposed accordingto a predetermined rule.

According to a third embodiment of the present disclosure, there isprovided an electronic apparatus including a solid-state imaging deviceincluding a pixel array and a pixel value correcting unit. The pixelarray includes a plurality of pixels, the plurality of pixels eachhaving one of a different exposure time and a different exposuresensitivity and being disposed according to a predetermined rule. Thepixel value correcting unit is configured to correct, among pixel valuesobtained from the plurality of pixels in the pixel array, a pixel valueof a pixel of the plurality of pixels that applies to a presetcondition, by using a pixel value of another pixel of the plurality ofpixels.

In the first to third embodiments of the present disclosure, among pixelvalues obtained from a plurality of pixels in a pixel array, a pixelvalue of a pixel of the plurality of pixels that applies to a presetcondition is corrected by using a pixel value of another pixel of theplurality of pixels, the plurality of pixels each having one of adifferent exposure time and a different exposure sensitivity and beingdisposed according to a predetermined rule.

According to the present disclosure, an image sensor capable ofcapturing an image in a high dynamic range can execute efficient defectcorrection processing.

These and other objects, features and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription of best mode embodiments thereof, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for describing a row exposure array in an SVE(Spatially Varying Exposure) method;

FIG. 2 is a diagram for describing a uniform exposure array in the SVEmethod;

FIG. 3 is a block diagram showing a configuration example of an imageprocessing device according to an embodiment of the present disclosure;

FIG. 4 is a block diagram showing a detailed configuration example of adefect correction processing unit;

FIG. 5 is a diagram showing an example of a processing unit area in therow exposure array;

FIG. 6 is a diagram for describing the processing of a flatnessdetermining unit of FIG. 4 in the row exposure array;

FIG. 7 is a diagram for describing the processing of a directiondetecting unit of FIG. 4 in the row exposure array;

FIG. 8 is a diagram for describing the processing of a defect correctingunit of FIG. 4 when the processing unit area is saturated;

FIG. 9 is a diagram showing an example of the processing unit area inthe uniform exposure array;

FIG. 10 is a diagram for describing the processing of the flatnessdetermining unit of FIG. 4 in the uniform exposure array;

FIG. 11 is a diagram for describing the processing of the flatnessdetermining unit of FIG. 4 when a pixel of interest is a G pixel in theuniform exposure array;

FIG. 12 is a diagram for describing the processing of the defectcorrecting unit of FIG. 4 when the processing unit area is not saturatedin the uniform exposure array;

FIG. 13 is a diagram for describing the processing of the directiondetecting unit of FIG. 4 when the processing unit area is saturated inthe uniform exposure array;

FIG. 14 is a diagram for describing the processing of the defectcorrecting unit of FIG. 4 when the processing unit area is saturated inthe uniform exposure array;

FIG. 15 is a diagram for describing a method of mixing a correctioncandidate value and the pixel value of the pixel of interest;

FIGS. 16A and 16B are each a diagram for describing a pixel area that isnecessary to select pixels having the same color and exposure time asthe pixel of interest and being in a vertical direction;

FIG. 17 is a flowchart for describing an example of defect correctionprocessing in the row exposure array;

FIG. 18 is a flowchart for describing an example of flatnessdetermination processing in an unsaturated state;

FIG. 19 is a flowchart for describing an example of defect determinationprocessing in a flat state;

FIG. 20 is a flowchart for describing an example of defect correctionprocessing in a flat state;

FIG. 21 is a flowchart for describing an example of direction detectionprocessing;

FIG. 22 is a flowchart for describing an example of defect determinationprocessing in a non-flat state;

FIG. 23 is a flowchart for describing an example of defect correctionprocessing in a non-flat state;

FIG. 24 is a flowchart for describing an example of defect determinationprocessing in a saturated state;

FIG. 25 is a flowchart for describing an example of defect correctionprocessing in a saturated state;

FIG. 26 is a flowchart for describing an example of defect correctionprocessing in the uniform exposure array;

FIG. 27 is a flowchart for describing an example of flatnessdetermination processing in a saturated state;

FIG. 28 is a flowchart for describing an example of defect determinationprocessing in a saturated and non-flat state;

FIG. 29 is a flowchart for describing an example of defect correctionprocessing in a saturated and non-flat state;

FIG. 30 is a flowchart for describing the example of the defectcorrection processing in the saturated and non-flat state;

FIG. 31 is a flowchart for describing the example of the defectcorrection processing in the saturated and non-flat state;

FIG. 32 is a diagram showing a configuration example of a solid-stateimaging device serving as a semiconductor device according to theembodiment of the present disclosure;

FIG. 33 is a diagram for describing a process flow of an image sensorhaving a laminate structure according to the embodiment of the presentdisclosure; and

FIG. 34 is a block diagram showing a configuration example of theimaging apparatus serving as an electronic apparatus to which theembodiment of the present disclosure is applied.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the technology to be disclosed herein willbe described with reference to the drawings.

FIGS. 1 and 2 are each a diagram showing an example of a plan view of apixel array of an image sensor that adopts an SVE (Spatially VaryingExposure) method. The SVE is a technique of obtaining an image in a highdynamic range by changing an exposure time in accordance with theposition of a pixel in the pixel array of the Bayer array, the image inthe high dynamic range being appropriately displayed from the pixels oflow luminance to the pixels of high luminance.

In the examples of FIGS. 1 and 2, pixels with hatching are each assumedas a short-time exposure pixel, and pixels without hatching are eachassumed as a long-time exposure pixel.

In the configurations of FIGS. 1 and 2, the exposure arrays differ fromeach other. Specifically, the array pattern of the short-time exposurepixels and the array pattern of the long-time exposure pixels aredifferent from each other.

In the configuration of FIG. 1, pixels in the two rows from the top areassumed as short-time exposure pixels, pixels in two rows immediatelybelow those two rows are assumed as long-time exposure pixels, pixels intwo rows further below are assumed as short-time exposure pixels, andpixels in two rows further below are assumed as long-time exposurepixels. Specifically, pixels having the same exposure time are regularlyarrayed on a row-by-row basis in the pixel array. The exposure arrayshown in FIG. 1 is also referred to as a row exposure array.

In the configuration of FIG. 2, the short-time exposure pixels andlong-time exposure pixels of respective colors are arrayed so as to bespatially uniform in the pixel array of the Bayer array. In thisexample, the short-time exposure pixels and the long-time exposurepixels are regularly arrayed as L-shaped groups of pixels, each groupincluding 4 pixels of R, G, G, and B. The exposure array shown in FIG. 2is also referred to as uniform exposure array.

FIG. 3 is a block diagram showing a configuration example of an imageprocessing device 10 according to the embodiment of the presentdisclosure. The image processing device 10 uses an image, whichcorresponds to an image signal output from the image sensor of the SVEmethod, as an input image, to generate and output an output image in ahigh dynamic range.

Further, as will be described later, the image processing device 10 ismounted to a so-called lower chip in a lamination-type image sensor andis formed integrally with a sensor chip of the image sensor of the SVEmethod.

In this example, the image processing device 10 includes a defectcorrection processing unit 21, a high dynamic range (HDR) combinationprocessing unit 22, and a gradation conversion processing unit 23.

The defect correction processing unit 21 is a block for correcting thepixel value of a defective pixel in the input image. The defectcorrection processing unit 21 corrects the pixel value of each pixelarrayed in the row exposure array or the uniform exposure array asappropriate.

The HDR combination processing unit 22 is a block for generating animage for a dark area and an image for a bright area from one inputimage, which is formed of pixels arrayed in the row exposure array orthe uniform exposure array, and combining those images. Specifically,the HDR combination processing unit 22 generates an image for a darkarea, in which the pixel values of the long-time exposure pixels areused to allow the dark area in the image to be clearly displayed, and animage for a bright area, in which the pixel values of the short-timeexposure pixels are used to allow the bright area in the image to beclearly displayed. Subsequently, the HDR combination processing unit 22combines those images for the dark area and for the bright area.

The gradation conversion processing unit 23 is a block for performinggradation conversion processing of adjusting the bit number of the pixelvalues of the image combined by the HDR combination processing unit 22,for example.

Next, the processing by the defect correction processing unit 21 will bedescribed in detail. FIG. 4 is a block diagram showing a detailedconfiguration example of the defect correction processing unit 21.

The defect correction processing unit 21 assumes each pixel, which isarrayed in an effective area of the pixel array of the image sensor, tobe a pixel of interest. In the case where the pixel of interest is adefective pixel, the defect correction processing unit 21 generates apixel signal of the pixel of interest by using the pixel signal ofanother pixel. In this example, the defect correction processing unit 21includes a gain correcting unit 41, a saturation determining unit 42, aflatness determining unit 43, a direction detecting unit 44, a defectdetermining unit 45, and a defect correcting unit 46.

Although the details are described later, the gain correcting unit 41 isa functional block that multiples by a gain so as to compare the pixelvalue of the short-time exposure pixel with the pixel value of thelong-time exposure pixel mainly in a situation where the long-timeexposure pixels and the short-time exposure pixels are mixed in thepixel array of the image sensor of the SVE method.

The saturation determining unit 42 is a functional block that mainlydetermines whether a predetermined area in the pixel array is saturatedor not due to the reception of strong light or the like. In the pixelarray of the image sensor of the SVE method, the long-time exposurepixels exist and thus when strong light is received, the pixel values ofthe long-time exposure pixels reach the maximum values and a state wherea correct pixel value corresponding to the amount of received light isnot obtained (saturated state) may occur.

The flatness determining unit 43 is a functional block that mainlydetermines whether a texture or the like is contained in an imagecorresponding to the light received in the predetermined area in thepixel array (specifically, the image is not flat) or not.

The direction detecting unit 44 is a functional block that detects thedirection of the texture in the case where the flatness determining unit43 determines that a texture or the like is contained in the imagecorresponding to the light received in the predetermined area in thepixel array.

The defect determining unit 45 is a functional block that mainlydetermines whether the pixel of interest is a defective pixel or not. Itshould be noted that the defective pixel is normally classified into ablown-out highlight (white-dot defective pixel) that constantly outputsa pixel signal having an extremely large value or into a blocked-upshadow (black-dot defective pixel) that constantly outputs a pixelsignal having an extremely small value.

The defect correcting unit 46 is a functional block that mainly correctsthe pixel value of the pixel of interest that is determined to be adefective pixel by the defect determining unit 45, by using the pixelvalue of another pixel.

It should be noted that the defect correction processing unit 21executes different processing between the row exposure array and theuniform exposure array, but here, the processing in the row exposurearray will be described first.

(Processing of Defect Correction Processing Unit 21 in Row ExposureArray)

The gain correcting unit 41 performs correction for adjusting adifference caused by the difference in exposure time on the pixel valuesof the input image.

Specifically, the pixel values of the input image correspond to imagesignals output from the image sensor of the SVE method, and thus thepixel values of the short-time exposure pixels and the pixel values ofthe long-time exposure pixels are mixed. For example, the gaincorrecting unit 41 multiples the pixel value of the short-time exposurepixel by a gain that is calculated as (exposure time of long-timeexposure pixel/exposure time of short-time exposure pixel). This allowsthe pixel value of the short-time exposure pixel to be corrected to apixel value corresponding to the long-time exposure. In other words, theprocessing by the gain correcting unit 41 normalizes the pixel values ofthe input image, in which the pixel values of the short-time exposurepixels and the pixel values of the long-time exposure pixels are mixed,based on the pixel values of the long-time exposure pixels.

The saturation determining unit 42 determines whether the pixel valuesare saturated or not for a processing unit area that is an area having apredetermined number of pixels centering on the pixel of interest.Specifically, the saturation determining unit 42 determines whether theprocessing unit area corresponds to a significantly bright area in theimage or not.

FIG. 5 is a diagram showing an example of the processing unit area. Inthis example, an area formed of 25 (=5×5) pixels in the pixel array ofthe Bayer array is the processing unit area. In the example of FIG. 5,the pixel of interest is indicated by an x mark and is a red (R)long-time exposure pixel in this example.

The saturation determining unit 42 specifies the number of pixels whosepixel values are the maximum values in the processing unit area formedof 25 pixels centering on the pixel of interest. Here, the pixel whosepixel value is the maximum value means, for example, a pixel having themaximum value that can be expressed in a predetermined bit number in thecase where the pixel value is expressed in digital data formed of thepredetermined bit number. For example, the maximum value is 255 whenexpressed in digital data of 8 bits. Specifically, here, when the amountof received light in a unit time is too large to output all the receivedlight as charge by photoelectric conversion, this state is referred toas saturation.

In the case where the number of pixels whose pixel values are themaximum values is equal to or larger than a preset threshold value, thesaturation determining unit 42 determines that the processing unit areais saturated. For example, in the case where the number of pixels whosepixel values are the maximum values is 3 (threshold value) or more, thesaturation determining unit 42 determines that the processing unit areais saturated. Here, the threshold value is set to 3. This is because, inconsideration of the case where the pixel of interest is a white-dotdefective pixel, whether the processing unit area is saturated or notcan be determined based on whether the two pixels other than the pixelof interest have the maximum values or not.

It should be noted that the processing from the flatness determiningunit 43 to the defect correcting unit 46 differs depending on the resultof the determination by the saturation determining unit 42. For example,in the case where the saturation determining unit 42 determines that theprocessing unit area is saturated, the determination by the flatnessdetermining unit 43 and the detection of a direction by the directiondetecting unit 44 are not performed. This is because it is difficult todetermine whether the saturated processing unit area is an areaoriginally containing the texture or not.

Here, the example of the processing from the flatness determining unit43 to the defect correcting unit 46 in the case where the saturationdetermining unit 42 first determines that the processing unit area isnot saturated will be described.

(Case where Processing Unit Area is not Saturated in Row Exposure Array)

The flatness determining unit 43 determines whether the processing unitarea is flat or not. For example, in the case where the processing unitarea is part of a texture in the image, the flatness determining unit 43determines that the processing unit area is not flat.

The flatness determining unit 43 compares the pixel value of the pixelof interest with the maximum or minimum value in the pixel values ofpixels having the same color as the pixel of interest in the processingunit area and determines whether the pixel value of the pixel ofinterest is larger or smaller than the maximum or minimum value. Forexample, in the case where the pixel of interest is an R long-timeexposure pixel, as shown in FIG. 6, the flatness determining unit 43specifies the maximum value or the minimum value of 8 R pixels (R₀ toR₇) indicated by circles in the processing unit area. In the case wherethe pixel value of the pixel of interest is larger than the maximumvalue of the pixel values of the R₀ to R₇, the pixel of interest isassumed as a candidate of the white-dot defective pixel and apredetermined flag or the like is set therefor. Further, in the casewhere the pixel value of the pixel of interest is smaller than theminimum value of the pixel values of the R₀ to R₇, the pixel of interestis assumed as a candidate of the black-dot defective pixel and apredetermined flag or the like is set therefor.

Additionally, the flatness determining unit 43 calculates an averagedeviation of the processing unit area. For example, in the example ofFIG. 6, the flatness determining unit 43 calculates an average value“ave_area” of the pixel values of the R₀ to R₇ by the followingexpression (1), and using the result of the calculation, calculates anaverage deviation “std_area” of the processing unit area by thefollowing expression (2).

$\begin{matrix}{{ave\_ area} = {\frac{1}{8}{\sum\limits_{i = 0}^{7}R_{i}}}} & (1) \\{{std\_ area} = {\frac{1}{8}{\sum\limits_{i = 0}^{7}{{{ave\_ area} - R_{i}}}}}} & (2)\end{matrix}$

In the case where the average deviation “std_area” calculated by theexpression (2) is smaller than a preset threshold value, the flatnessdetermining unit 43 determines that the processing unit area is flat. Inthe case where the average deviation “std_area” is equal to or largerthan the preset threshold value, the flatness determining unit 43determines that the processing unit area is not flat.

As described above, in the case where the saturation determining unit 42determines that the processing unit area is saturated, the determinationby the flatness determining unit 43 is not performed.

In the case where it is determined that the processing unit area is notflat, the direction detecting unit 44 determines a direction of atexture (pattern) included in the processing unit area.

The direction detecting unit 44 extracts a direction detection area. Thedirection detection area is formed of 35 (=7×5) pixels and is obtainedby expanding the right and left ends of the processing unit area by onepixel. The direction detecting unit 44 calculates 15 absolute values ofdifferences between adjacent pixels having the same color in thedirection detection area. At that time, as shown in FIG. 7, thedirection detecting unit 44 calculates 15 absolute values of differencesbetween adjacent pixels having the same color in each of 4 directions,that is, a horizontal direction, a vertical direction, a +45° direction,and a −45° direction.

FIG. 7 is a diagram showing, in the upper left part, an example of thecalculation of the absolute values of differences between the pixelshaving the same color and being adjacent in the horizontal direction inthe direction detection area. As indicated by horizontal arrows in theupper left part of FIG. 7, the 15 sets of pixels, each set having thesame color, are shown.

FIG. 7 is the diagram showing, in the upper right part, an example ofthe calculation of the absolute values of differences between the pixelshaving the same color and being adjacent in the vertical direction inthe direction detection area. As indicated by vertical arrows in theupper right part of FIG. 7, the 15 sets of pixels, each set having thesame color, are shown.

FIG. 7 is the diagram showing, in the lower left part, an example of thecalculation of the absolute values of differences between the pixelshaving the same color and being adjacent in the +45° direction in thedirection detection area. As indicated by arrows in the +45° directionin the lower left part of FIG. 7, the 15 sets of pixels, each set havingthe same color, are shown.

FIG. 7 is the diagram showing, in the lower right part, an example ofthe calculation of the absolute values of differences between the pixelshaving the same color and being adjacent in the −45° direction in thedirection detection area. As indicated by arrows in the −45° directionin the lower right part of FIG. 7, the 15 sets of pixels, each sethaving the same color, are shown.

Next, in order to eliminate the influence of the defective pixel in thearea, the direction detecting unit 44 selects 11 low-order absolutevalues of differences from the 15 absolute values of the differences ineach of the directions. Specifically, since the absolute values of thedifferences using the pixel value of a defective pixel do not makesense, 4 high-order absolute values of the differences are eliminated inorder to remove the absolute values of the differences using the pixelvalue of a defective pixel. Here, in consideration of the pixel ofinterest and another pixel (2 pixels in total) that are defective pixelsin the direction detection area, 4 high-order absolute values of thedifferences are eliminated.

The direction detecting unit 44 calculates an average value of the 11absolute values of the differences in each of the directions.

Here, the average value of the 11 absolute values of the differences inthe horizontal direction and the average value of the 11 absolute valuesof the differences in the vertical direction are each multiplied by √2.This is because a distance between adjacent pixels in the horizontaldirection or the vertical direction is smaller than a distance betweenadjacent pixels in the +45° direction or the −45° direction and suchinfluence is eliminated.

Additionally, the direction detecting unit 44 selects the minimum valueof the 4 average values corresponding to the respective directions anddetects a direction corresponding to the minimum value as the directionof the texture.

As described above, in the case where the saturation determining unit 42determines that the processing unit area is saturated, the detection ofthe direction by the direction detecting unit 44 is not performed.

The defect determining unit 45 determines whether the pixel of interestis a defective pixel or not.

The defect determining unit 45 first calculates a threshold value fordefect determination. It should be noted that the defect determiningunit 45 calculates a threshold value based on the result of thedetermination by the flatness determining unit 43. In the case where theflatness determining unit 43 determines that the processing unit area isflat, the defect determining unit 45 calculates the threshold value bythe following expression (3).

Th1=a×√{square root over (ave_area)}+b  (3)

It should be noted that “a” and “b” in the expression (3) are parametersand are changed in accordance with the magnitude of the average value“ave_area” of the pixel values of the R₀ to R_(7.)

On the other hand, in the case where the flatness determining unit 43determines that the processing unit area is not flat, the defectdetermining unit 45 calculates the threshold value by the followingexpression (4).

Th2=a×√{square root over (ave_area)}+b×grad+c  (4)

It should be noted that “a”, “b”, and “c” in the expression (4) areparameters and are changed in accordance with the magnitude of a minimumvalue “grad” used for detecting the direction of the texture.

In the case where the processing unit area is flat and the pixel ofinterest is a candidate of the white-dot defective pixel, the defectdetermining unit 45 determines whether the absolute value of adifference between the maximum value of the R₀ to the R₇ of FIG. 6 andthe pixel of interest is equal to or larger than the threshold valuedescribed above. In the case where the absolute value of a differencebetween the maximum value of the R₀ to R₇ and the pixel of interest isequal to or larger than the threshold value described above, the defectdetermining unit 45 determines that the pixel of interest is a white-dotdefective pixel. In the case where the absolute value of a differencebetween the maximum value of the R₀ to R₇ and the pixel of interest issmaller than the threshold value, the defect determining unit 45determines that the pixel of interest is not a defective pixel.

Further, in the case where the processing unit area is flat and thepixel of interest is a candidate of the black-dot defective pixel, thedefect determining unit 45 determines whether the absolute value of adifference between the minimum value of the R₀ to R₇ of FIG. 6 and thepixel of interest is equal to or larger than the threshold valuedescribed above. In the case where the absolute value of a differencebetween the minimum value of the R₀ to R₇ and the pixel of interest isequal to or larger than the threshold value described above, the defectdetermining unit 45 determines that the pixel of interest is a black-dotdefective pixel. In the case where the absolute value of a differencebetween the minimum value of the R₀ to R₇ and the pixel of interest issmaller than the threshold value, the defect determining unit 45determines that the pixel of interest is not a defective pixel.

Additionally, in the case where the processing unit area is not flat,the defect determining unit 45 determines whether a difference betweenthe pixel value of the pixel of interest and the average value of thepixel values of pixels (for example, R₀ to R₇ in FIG. 6) having the samecolor in the processing unit area is equal to or larger than thethreshold value described above.

In the case where the difference is equal to or larger than thethreshold value, the defect determining unit 45 determines that thepixel of interest is a defective pixel. In the case where the differenceis smaller than the threshold value, the defect determining unit 45determines that the pixel of interest is not a defective pixel.Additionally, in the case where it is determined that the pixel ofinterest is a defective pixel, when the pixel value of the pixel ofinterest is larger than the average value of the pixel values of thepixels having the same color in the processing unit area, it isdetermined that the pixel of interest is a white-dot defective pixel,and when the pixel value of the pixel of interest is smaller than theaverage value of the pixel values of the pixels having the same color inthe processing unit area, it is determined that the pixel of interest isa black-dot defective pixel.

In the case where the processing unit area is flat and when the defectdetermining unit 45 determines that the pixel of interest is a defectivepixel, the defect correcting unit 46 performs correction by replacingthe pixel value of the pixel of interest with the pixel value of a pixelhaving the same color and exposure time as the pixel of interest in theprocessing unit area.

For example, in the case where the pixel of interest is determined to bea white-dot defective pixel in FIG. 6, the defect correcting unit 46replaces the pixel value of the pixel of interest with a larger one ofthe pixel values of the R₃ and the R₄ in FIG. 6.

Further, for example, in the case where the pixel of interest isdetermined to be a black-dot defective pixel in FIG. 6, the defectcorrecting unit 46 replaces the pixel value of the pixel of interestwith a smaller one of the pixel values of the R₃ and R₄ in FIG. 6.

Additionally, in the case where the processing unit area is not flat andwhen the defect determining unit 45 determines that the pixel ofinterest is a defective pixel, the defect correcting unit 46 replacesthe pixel value based on the direction of the texture that is detectedby the direction detecting unit 44.

Specifically, in the case where the direction of the texture is thehorizontal direction, when it is determined that the pixel of interestis a white-dot defective pixel, the pixel value of the pixel of interestis replaced with a larger one of the pixel values of the R₃ and R₄ ofFIG. 6, and when it is determined that the pixel of interest is ablack-dot defective pixel, the pixel value of the pixel of interest isreplaced with a smaller one of the pixel values of the R₃ and R₄ of FIG.6. Further, in the case where the direction of the texture is thevertical direction, when it is determined that the pixel of interest isa white-dot defective pixel, the pixel value of the pixel of interest isreplaced with a larger one of the pixel values of the R₁ and R₆ of FIG.6, and when it is determined that the pixel of interest is a black-dotdefective pixel, the pixel value of the pixel of interest is replacedwith a smaller one of the pixel values of the R₁ and R₆ of FIG. 6.Additionally, in the case where the direction of the texture is the +45°direction, when it is determined that the pixel of interest is awhite-dot defective pixel, the pixel value of the pixel of interest isreplaced with a larger one of the pixel values of the R₂ and R₅ of FIG.6, and when it is determined that the pixel of interest is a black-dotdefective pixel, the pixel value of the pixel of interest is replacedwith a smaller one of the pixel values of the R₂ and R₅ of FIG. 6.Additionally, in the case where the direction of the texture is the −45°direction, when it is determined that the pixel of interest is awhite-dot defective pixel, the pixel value of the pixel of interest isreplaced with a larger one of the pixel values of the R₀ and R₇ of FIG.6, and when it is determined that the pixel of interest is a black-dotdefective pixel, the pixel value of the pixel of interest is replacedwith a smaller one of the pixel values of the R₀ and R₇ of FIG. 6.

It should be noted that the pixels of the R₀ to R₂ and the pixels of theR₅ to R₇ in FIG. 6 have exposure time that is different from theexposure time of the pixel of interest, but as described above, thepixel values in the processing unit area are normalized based on thepixel values of the long-time exposure pixels by the processing of thegain correcting unit 41. Thus, the pixels of the R₀ to R₂ and the pixelsof the R₅ to R₇ can be used for the replacement.

In the case where the pixel of interest is a short-time exposure pixel,however, the defect correcting unit 46 multiplies the pixel value, whichis replaced based on the direction of the texture, by a gain that iscalculated as (exposure time of short-time exposure pixel/exposure timeof long-time exposure pixel), and thus eliminates the influence of thecorrection by the gain correcting unit 41.

In the case where the saturation determining unit 42 determines that theprocessing unit area is not saturated, whether the pixel of interest isa defective pixel or not is determined in the manner described above,and the pixel value in the case where the pixel of interest is adefective pixel is corrected.

(Case where Processing Unit Area is Saturated in Row Exposure Array)

Meanwhile, in the case where the saturation determining unit 42determines that the processing unit area is saturated, the defectdetermining unit 45 calculates a dynamic range of 9 pixels that arearranged in the horizontal direction while centering on the pixel ofinterest. For example, an area formed of 9 (1×9) pixels shown in FIG. 8is extracted as a dynamic range detection area. It should be noted thatwhen the saturation determining unit 42 determines that the processingunit area is saturated, a calculation using a pixel value before thecorrection by the gain correcting unit 41 is performed.

For pixels having the same color as the pixel of interest and beingindicated by circles in FIG. 8 (in this example, R pixels), the defectdetermining unit 45 calculates 4 absolute values of differences betweenthe pixel values of those pixels that are adjacent as indicated byarrows in FIG. 8, and calculates a dynamic range in the 4 absolutevalues of the differences. Further, the defect determining unit 45calculates a dynamic range of the pixel values of pixels having a colorthat is different from the color of the pixel of interest (in thisexample, G pixels).

In this example, the case where the pixel of interest is the R pixel hasbeen described, but in the case where the pixel of interest is a Gpixel, the R pixel is selected as a pixel having a color that isdifferent from the color of the pixel of interest. Further, in the casewhere the pixel of interest is a B pixel, the G pixel is selected as apixel having a color that is different from the color of the pixel ofinterest.

Subsequently, the defect determining unit 45 calculates a square root ofthe average value of the pixel values of pixels having the same color asthe pixel of interest and being indicated by circles in FIG. 8, andmultiplies by a parameter or adding a parameter as appropriate, to setthe resultant value as a threshold value for the defect determination.Since first to third conditions are used for the threshold value asdescribed later, three threshold values are calculated with changedparameters.

In the case where all of the following three conditions are satisfied,the defect determining unit 45 determines that the pixel of interest isa white-dot defective pixel.

The first condition is as follows: the pixel value of the pixel ofinterest is larger than the second-largest pixel value in the pixelvalues of the 4 pixels having the same color and being indicated by thecircles in FIG. 8, and a difference between the pixel value of the pixelof interest and the second-largest pixel value in the pixel values ofthe 4 pixels indicated by the circles in FIG. 8 is larger than thethreshold value.

In consideration of the case where defective pixels having the samecolor alternately exist, like the order of a defective pixel, anon-defective pixel, a defective pixel, . . . , for example, adifference between the pixel value of the pixel of interest and thelargest pixel value is not compared with the threshold value, and thedifference between the pixel value of the pixel of interest and thesecond-largest pixel value is compared with the threshold value.

The second condition is as follows: the dynamic range in the absolutevalues of differences between the pixel values of the adjacent pixelshaving the same color as the pixel of interest and being indicated bythe circles in FIG. 8 is larger than the threshold value.

The third condition is as follows: the dynamic range of the pixel valuesof pixels having a color that is different from the color of the pixelof interest shown in FIG. 8 (in FIG. 8, G pixels) is smaller than thethreshold value.

In the case where all of the first to third conditions described aboveare satisfied, it is determined that the pixel of interest is awhite-dot defective pixel.

Further, in the case where all of the following three conditions aresatisfied, the defect determining unit 45 determines that the pixel ofinterest is a black-dot defective pixel.

The first condition is as follows: the pixel value of the pixel ofinterest is smaller than the second-smallest pixel value in the pixelvalues of the 4 pixels having the same color and being indicated by thecircles in FIG. 8, and a difference between the pixel value of the pixelof interest and the second-smallest pixel value in the pixel values ofthe 4 pixels indicated by the circles in FIG. 8 is larger than thethreshold value.

The second and third conditions are the same as those of the white-dotdefective pixel, and thus its detailed description will be omitted.

In the case where all of the first to third conditions described aboveare satisfied, it is determined that the pixel of interest is ablack-dot defective pixel.

In the case where the pixel of interest is a white-dot defective pixel,the defect correcting unit 46 replaces the pixel value of the pixel ofinterest with the second-largest pixel value in the pixel values of the4 pixels indicated by the circles in FIG. 8. Further, in the case wherethe pixel of interest is a black-dot defective pixel, the defectcorrecting unit 46 replaces the pixel value of the pixel of interestwith the second-smallest pixel value in the pixel values of the 4 pixelsindicated by the circles in FIG. 8.

In the case where the saturation determining unit 42 determines that theprocessing unit area is saturated, whether the pixel of interest is adefective pixel or not is determined in the manner described above, andthe pixel value in the case where the pixel of interest is a defectivepixel is corrected.

In such a manner, the defect correction processing unit 21 corrects thepixel value of the defective pixel.

Next, description will be given on the processing of the defectcorrection processing unit 21 in the case of the uniform exposure array.The case of the uniform exposure array is different from the case of therow exposure array in that the determination by the flatness determiningunit 43 and the detection of a direction by the direction detecting unit44 are performed also when the processing unit area is saturated.

(Processing of Defect Correction Processing Unit 21 in Uniform ExposureArray)

As in the row exposure array, the gain correcting unit 41 performscorrection for adjusting a difference caused by the difference inexposure time on the pixel values of the input image.

As in the row exposure array, the saturation determining unit 42determines whether the pixel values are saturated or not for aprocessing unit area that is an area having a predetermined number ofpixels centering on the pixel of interest. Specifically, the saturationdetermining unit 42 determines whether the processing unit areacorresponds to a significantly bright area in the image or not.

FIG. 9 is a diagram showing an example of the processing unit area inthe uniform exposure array. In this example, an area formed of 25 (=5×5)pixels in the pixel array of the Bayer array is the processing unitarea. In the example of FIG. 9, the pixel of interest is indicated by anx mark and is a red (R) long-time exposure pixel in this example.

In the case where the number of pixels whose pixel values are themaximum values is equal to or larger than a preset threshold value, thesaturation determining unit 42 determines that the processing unit areais saturated. For example, in the case where the number of pixels whosepixel values are the maximum values is 3 (threshold value) or more, thesaturation determining unit 42 determines that the processing unit areais saturated.

It should be noted that also in the uniform exposure array, theprocessing from the flatness determining unit 43 to the defectcorrecting unit 46 differs depending on the result of the determinationby the saturation determining unit 42.

Here, the example of the processing from the flatness determining unit43 to the defect correcting unit 46 in the case where the saturationdetermining unit 42 first determines that the processing unit area isnot saturated will be described.

(Case where Processing Unit Area is not Saturated in Uniform ExposureArray)

The flatness determining unit 43 determines whether the processing unitarea is flat or not. For example, in the case where the processing unitarea is part of a texture in the image, the flatness determining unit 43determines that the processing unit area is not flat.

As in the row exposure array, the flatness determining unit 43 comparesthe pixel value of the pixel of interest with the maximum or minimumvalue in the pixel values of pixels having the same color as the pixelof interest in the processing unit area and determines whether the pixelvalue of the pixel of interest is larger or smaller than the maximum orminimum value. For example, in the case where the pixel of interest isan R long-time exposure pixel, as shown in FIG. 10, the flatnessdetermining unit 43 specifies the maximum value or the minimum value of8 R pixels (R₀ to R₇) indicated by circles in the processing unit area.In the case where the pixel value of the pixel of interest is largerthan the maximum value in the pixel values of the R₀ to R₇, the pixel ofinterest is assumed as a candidate of the white-dot defective pixel anda predetermined flag or the like is set therefor. Further, in the casewhere the pixel value of the pixel of interest is smaller than theminimum value in the pixel values of the R₀ to R₇, the pixel of interestis assumed as a candidate of the black-dot defective pixel and apredetermined flag or the like is set therefor.

Additionally, as in the row exposure array, the flatness determiningunit 43 calculates an average deviation of the processing unit area.

In the case where the calculated average deviation “std_area” is smallerthan a preset threshold value, the flatness determining unit 43determines that the processing unit area is flat. In the case where theaverage deviation “std_area” is equal to or larger than the presetthreshold value, the flatness determining unit 43 determines that theprocessing unit area is not flat.

In the uniform exposure array, however, when the pixel of interest is aG pixel, it is assumed that pixels having the same color as the pixel ofinterest are disposed in the processing unit area as shown in FIG. 11.Specifically, in the processing unit area, all the pixels having thesame color as the pixel of interest have the same exposure time (in thisexample, short-time exposure pixels).

Consequently, when the pixel of interest is a G pixel, a calculationusing a pixel value before correction by the gain correcting unit 41 isperformed. For that reason, in the uniform exposure array, threethreshold values, that is, a threshold value for an R or B pixel, athreshold value for a G long-time exposure pixel, and a threshold valuefor a G short-time exposure pixel, are prepared as threshold values usedfor determining whether the processing unit area is flat or not.

In the case where it is determined that the processing unit area is notflat, as in the row exposure array, the direction detecting unit 44determines a direction of a texture (pattern) included in the processingunit area.

In each of the case where the processing unit area is flat and the casewhere the processing unit area is not flat, as in the row exposurearray, the defect determining unit 45 determines whether the pixel ofinterest is a defective pixel or not. When a threshold value for defectdetermination is calculated, however, the defect determining unit 45calculates three threshold values, that is, a threshold value for an Ror B pixel, a threshold value for a G long-time exposure pixel, and athreshold value for a G short-time exposure pixel.

As in the row exposure array, in the case where the processing unit areais flat and when the defect determining unit 45 determines that thepixel of interest is a defective pixel, the defect correcting unit 46performs correction by replacing the pixel value of the pixel ofinterest with the pixel value of a pixel having the same color andexposure time as the pixel of interest in the processing unit area.

In the uniform exposure array, when the pixel of interest is an R pixel,the R₀, R₂, R₅, or R₇ in FIG. 10 is used as a pixel to be replaced, andwhen the pixel of interest is a G pixel, G₀ to G₇ in FIG. 11 is used asa pixel to be replaced. It should be noted that when the pixel ofinterest is a B pixel, as in the case where the pixel of interest is anR pixel, a pixel having the same color and exposure time as the pixel ofinterest in the processing unit area is used.

Additionally, as in the row exposure array, in the case where theprocessing unit area is not flat and when the defect determining unit 45determines that the pixel of interest is a defective pixel, the defectcorrecting unit 46 replaces the pixel value based on the direction ofthe texture that is detected by the direction detecting unit 44.

In the case where the saturation determining unit 42 determines that theprocessing unit area is not saturated, whether the pixel of interest isa defective pixel or not is determined in the manner described above,and the pixel value in the case where the pixel of interest is adefective pixel is corrected.

(Case where Processing Unit Area is Saturated in Uniform Exposure Array)

Meanwhile, in the processing of correcting a defective pixel related tothe uniform exposure array, the processing by the flatness determiningunit 43 and the processing by the direction detecting unit 44 areperformed also when the saturation determining unit 42 determines thatthe processing unit area is saturated.

In the case where the pixel of interest is a G pixel, as in the rowexposure array, the flatness determining unit 43 determines whether theprocessing unit area is flat or not.

On the other hand, in the case where the pixel of interest is an R or Bpixel, the flatness determining unit 43 determines whether the pixelvalue of the pixel of interest is larger or smaller than the maximum orminimum value in the pixel values of 4 pixels having the same color andexposure time as the pixel of interest and being indicated by circles inFIG. 12, for example. In the case where the pixel value of the pixel ofinterest is larger than the maximum value in the pixel values of the 4pixels, the pixel of interest is assumed as a candidate of the white-dotdefective pixel and a predetermined flag or the like is set therefor.Further, in the case where the pixel value of the pixel of interest issmaller than the minimum value in the pixel values of the 4 pixels, thepixel of interest is assumed as a candidate of the black-dot defectivepixel and a predetermined flag or the like is set therefor.

It should be noted that FIG. 12 shows as an example the case where thepixel of interest is the R long-time exposure pixel, but the same holdstrue for the case where the pixel of interest is an R short-timeexposure pixel, a B long-time exposure pixel, or a B short-time exposurepixel.

Additionally, the flatness determining unit 43 calculates an averagedeviation of the processing unit area. For example, in the example ofFIG. 10, the flatness determining unit 43 calculates an average value“ave_area” of the pixel values of the R₀, R₂, R₅, and R₇ by thefollowing expression (5), and using the result of the calculation,calculates an average deviation “std_area” of the processing unit areaby the following expression (6).

$\begin{matrix}{{ave\_ area} = {\frac{1}{4}{\sum\limits_{i = 0}^{3}R_{i}}}} & (5) \\{{std\_ area} = {\frac{1}{4}{\sum\limits_{i = 0}^{3}{{{ave\_ area} - R_{i}}}}}} & (6)\end{matrix}$

In the case where the average deviation “std_area” calculated by theexpression (6) is smaller than a preset threshold value, the flatnessdetermining unit 43 determines that the processing unit area is flat. Inthe case where the average deviation “std_area” is equal to or largerthan the preset threshold value, the flatness determining unit 43determines that the processing unit area is not flat.

It should be noted that in the uniform exposure array, when theprocessing unit area is saturated, 4 threshold values, that is, athreshold value for an R or B long-time exposure pixel, a thresholdvalue for an R or B short-time exposure pixel, a threshold value for a Glong-time exposure pixel, and a threshold value for a G short-timeexposure pixel, are prepared as threshold values used for determiningwhether the processing unit area is flat or not.

In the case where it is determined that the processing unit area is notflat, the direction detecting unit 44 determines a direction of atexture (pattern) included in the processing unit area.

In this case, since it is thought that the long-time exposure pixels aresaturated, the direction detection using only G short-time exposurepixels is performed.

The direction detecting unit 44 extracts a direction detection area thatis formed of 35 (=7×5) pixels and is obtained by expanding the right andleft ends of the processing unit area by one pixel. The directiondetecting unit 44 calculates absolute values of differences betweenadjacent G pixels in the direction detection area. At that time, asshown in FIG. 13, the direction detecting unit 44 calculates absolutevalues of differences between adjacent G pixels in each of 4 directions,that is, a horizontal direction, a vertical direction, a +45° direction,and −45° direction.

FIG. 13 is a diagram showing, in the upper left part, an example of thecalculation of the absolute values of differences between G pixels thatare adjacent in the horizontal direction in the direction detectionarea. As indicated by horizontal arrows in the upper left part of FIG.13, 9 sets of G pixels are shown.

FIG. 13 is the diagram showing, in the upper right part, an example ofthe calculation of the absolute values of differences between G pixelsthat are adjacent in the vertical direction in the direction detectionarea. As indicated by vertical arrows in the upper right part of FIG.13, 8 sets of G pixels are shown.

FIG. 13 is the diagram showing, in the lower left part, an example ofthe calculation of the absolute values of differences between G pixelsthat are adjacent in the +45° direction in the direction detection area.As indicated by arrows in the +45° direction in the lower left part ofFIG. 13, six sets of G pixels are shown.

FIG. 13 is the diagram showing, in the lower right part, an example ofthe calculation of the absolute values of differences between G pixelsthat are adjacent in the −45° direction in the direction detection area.As indicated by arrows in the −45° direction in the lower right part ofFIG. 13, six sets of G pixels are shown.

Next, in order to eliminate the influence of the defective pixel in thearea, the direction detecting unit 44 selects 4 low-order absolutevalues of differences from the absolute values of the differences ineach of the directions. Specifically, since the absolute values of thedifferences using the pixel value of a defective pixel do not makesense, high-order absolute values of the differences are eliminated inorder to remove the absolute values of the differences using the pixelvalue of a defective pixel.

The direction detecting unit 44 calculates an average value of the 4absolute values of the differences in each of the directions.

Here, the average value of the 4 absolute values of the differences inthe horizontal direction and the average value of the 4 absolute valuesof the differences in the vertical direction are each multiplied by A/2.This is because a distance between adjacent pixels in the horizontaldirection or the vertical direction is smaller than a distance betweenadjacent pixels in the +45° direction or the −45° direction and suchinfluence is eliminated.

Additionally, the direction detecting unit 44 selects the minimum valueof the 4 average values corresponding to the respective directions anddetects a direction corresponding to the minimum value as the directionof the texture.

In the example of FIG. 13, in the case where the pixel of interest is aG short-time exposure pixel or a B pixel, when an area formed of 35(=7×5) pixels in which the number of G short-time exposure pixelsbecomes the largest is extracted, the position of the pixel of interestdoes not come to the center of the area. In such a case, the result ofthe direction detection with a pixel located on the left of the pixel ofinterest being at the center is used.

In the case where the processing unit area is flat, as in the rowexposure array, the defect determining unit 45 determines whether thepixel of interest is a defective pixel or not. In the uniform exposurearray, however, when a threshold value for defect determination iscalculated, the defect determining unit 45 calculates three thresholdvalues, that is, a threshold value for an R or B pixel, a thresholdvalue for a G long-time exposure pixel, and a threshold value for a Gshort-time exposure pixel.

Further, in the case where the processing unit area is not flat, thedefect determining unit 45 determines whether the pixel of interest is adefective pixel or not as follows.

Specifically, when a threshold value for defect determination iscalculated, the defect determining unit 45 calculates a threshold valueby using the average value of pixels having the same color and exposuretime as the pixel of interest in the processing unit area and using theaverage value corresponding to the direction detected by the directiondetecting unit 44 (the average value of the 4 absolute values of thedifferences). For example, in the case where the pixel of interest is anR or B pixel, the defect determining unit 45 calculates a thresholdvalue by using a square root of the average value of the pixel values ofthe 4 pixels, which are located at positions expressed in white andindicated by the circles in FIG. 10, and using the average valuecorresponding to the direction detected by the direction detecting unit44. On the other hand, in the case where the pixel of interest is a Gpixel, the defect determining unit 45 calculates a threshold value byusing a square root of the average value of the pixel values of the 8pixels, which are located at positions indicated by the circles in FIG.11, and using the average value corresponding to the direction detectedby the direction detecting unit 44.

However, when the threshold value for the defect determination iscalculated, the defect determining unit 45 calculates three thresholdvalues, that is, a threshold value for an R or B pixel, a thresholdvalue for a G long-time exposure pixel, and a threshold value for a Gshort-time exposure pixel.

In the case where the pixel of interest is the R or B pixel and when adifference between the pixel value of the pixel of interest and theaverage value of the pixel values of the 4 pixels, which are located atpositions expressed in white and indicated by the circles in FIG. 10, isequal to or larger than the threshold value, the defect determining unit45 determines that the pixel of interest is a defective pixel. Further,in the case where the pixel of interest is a G pixel and when adifference between the pixel value of the pixel of interest and theaverage value of the pixel values of the 8 pixels, which are located atpositions indicated by the circles in FIG. 11, is equal to or largerthan the threshold value, the defect determining unit 45 determines thatthe pixel of interest is a defective pixel.

As in the row exposure array, in the case where the processing unit areais flat and when the defect determining unit 45 determines that thepixel of interest is a defective pixel, the defect correcting unit 46performs correction by replacing the pixel value of the pixel ofinterest with the pixel value of a pixel having the same color andexposure time as the pixel of interest in the processing unit area.

On the other hand, in the case where the processing unit area is notflat, the defect correcting unit 46 performs different processingdepending on whether the pixel of interest is the G pixel or the pixelof interest is the R or B pixel.

In the case where the pixel of interest is the G pixel, the defectcorrecting unit 46 selects two pixels from the 8 pixels located atpositions indicated by the circles in FIG. 11, based on the direction ofthe texture that is detected by the direction detecting unit 44.

Specifically, in the case where the direction of the texture is thehorizontal direction, the G₃ and the G₄ in FIG. 11 are selected.Further, in the case where the direction of the texture is the verticaldirection, the G₁ and the G₆ in FIG. 11 are selected. Furthermore, inthe case where the direction of the texture is the +45° direction, theG₂ and the G₅ in FIG. 11 are selected. Additionally, in the case wherethe direction of the texture is the −45° direction, the G₀ and the G₇ inFIG. 11 are selected.

In the case where it is determined that the pixel of interest is awhite-dot defective pixel, the defect correcting unit 46 replaces thepixel value of the pixel of interest with a larger one of the pixelvalues of the two pixels selected as described above. Further, in thecase where it is determined that the pixel of interest is a black-dotdefective pixel, the defect correcting unit 46 replaces the pixel valueof the pixel of interest with a smaller one of the pixel values of thetwo pixels selected as described above.

In the case where the pixel of interest is the R or B pixel, the defectcorrecting unit 46 selects two pixels from the 8 pixels located atpositions indicated by circles in FIG. 14 for example, based on thedirection of the texture that is detected by the direction detectingunit 44.

Specifically, in the case where the direction of the texture is the +45°direction, the R₂ and the R₅ of FIG. 14 are selected. In the case wherethe direction of the texture is the −45° direction, the R₀ and the R₇ ofFIG. 14 are selected.

In the case where the direction of the texture is the horizontaldirection, the R₃ and the R₄ of FIG. 14 are selected. In the uniformexposure array, a pixel that is closest to the pixel of interest in thehorizontal direction and has the same color as the pixel of interest isa pixel having exposure time that is different from the exposure time ofthe pixel of interest. So, it is necessary to select pixels from adirection detection area formed of 45 (=9×5) pixels, which is obtainedby further expanding the direction detection area formed of 35 (=7×5)pixels.

In the case where the direction of the texture is the verticaldirection, the pixel values of pixels at two positions in the verticaldirection are calculated by linear interpolation. Specifically, avirtual pixel having a pixel value calculated as (R₀+R₂)/2 and a virtualpixel having a pixel value calculated as (R₅+R₇)/2 are selected. This isbecause, in the uniform exposure array, pixels having the same color andexposure time as the pixel of interest do not exist in the verticaldirection in the area corresponding to five rows.

In the case where it is determined that the pixel of interest is awhite-dot defective pixel, the defect correcting unit 46 replaces thepixel value of the pixel of interest with a larger one of the pixelvalues of the two pixels selected as described above. Further, in thecase where it is determined that the pixel of interest is a black-dotdefective pixel, the defect correcting unit 46 replaces the pixel valueof the pixel of interest with a smaller one of the pixel values of thetwo pixels selected as described above.

However, in the case where the pixel of interest is the R or B pixel andthe direction of the texture is the vertical direction, since thereplacement with the pixel value of the virtual pixel is performed asdescribed above, the defect correcting unit 46 not totally but partiallyreplaces the pixel value of the pixel of interest with the pixel valueof the virtual pixel.

In the case where the pixel of interest is the R or B pixel and thedirection of the texture is the vertical direction, the defectcorrecting unit 46 calculates, together with the pixel values of the twovirtual pixels described above, an average value (in the case of FIG.14, (R₁+R₆)/2) of the pixels having the same color in the verticaldirection (in this case, the virtual pixels obtained by linearinterpolation). It should be noted that the pixels having the same colorand being in the vertical direction have exposure time that is differentfrom the exposure time of the pixel of interest. So, the pixel values ofthe short-time exposure pixels are previously corrected by beingmultiplied by a gain obtained by (exposure time of long-time exposurepixel/exposure time of short-time exposure pixel).

Next, a difference between a pixel value to be replaced, which is one ofthe pixel values of the virtual pixels and is referred to as correctioncandidate value, and the average value of the pixel values of the pixelshaving the same color in the vertical direction is calculated.Subsequently, in accordance with the magnitude of the difference, thepixel value of the pixel of interest is mixed to the correctioncandidate value.

FIG. 15 is a diagram for describing a method of mixing the correctioncandidate value and the pixel value of the pixel of interest. In FIG.15, the horizontal axis indicates the difference between the correctioncandidate value and the average value of the pixel values of the pixelshaving the same color in the vertical direction, and the vertical axisindicates a mixing ratio of the pixel value of the pixel of interest.

As shown in FIG. 15, a threshold value X and a threshold value Y arepreviously set, and in the case where the difference between thecorrection candidate value and the average value of the pixel values ofthe pixels having the same color in the vertical direction is smallerthan the threshold value X, the mixing ratio of the pixel value of thepixel of interest is set to 0. Further, in the case where the differencebetween the correction candidate value and the average value of thepixel values of the pixels having the same color in the verticaldirection is equal to or larger than the threshold value Y, the mixingratio of the pixel value of the pixel of interest is set to 1.Furthermore, in the case where the difference between the correctioncandidate value and the average value of the pixel values of the pixelshaving the same color in the vertical direction is equal to or largerthan the threshold value X and smaller than the threshold value Y, themixing ratio of the pixel value of the pixel of interest is changed inaccordance with the magnitude of the difference between the correctioncandidate value and the average value of the pixel values of the pixelshaving the same color in the vertical direction.

In the case where the difference between the correction candidate valueand the average value of the pixel values of the pixels having the samecolor in the vertical direction is large, it is thought that the imagein the processing unit area includes the R or B pixels having acomponent near a Nyquist frequency. In such a case, when a correction isperformed using the pixel value of the virtual pixel that is obtained bylinear interpolation by use of adjacent pixels, a color that does notoriginally exist may be added or artifacts may occur.

As described above, in the case where the pixel of interest is the R orB pixel and the direction of the texture is the vertical direction, thepixel value of the pixel of interest is partially replaced with thepixel value of the virtual pixel. This can allow the occurrence of theaddition of color or artifacts to be suppressed.

Additionally, in the present disclosure, in the case where the pixel ofinterest is the R or B pixel and the direction of the texture is thevertical direction as described above, the pixel value of the pixel ofinterest is partially replaced with the pixel value of the virtualpixel. This can allow the capacity of a memory necessary in the defectcorrection processing to be suppressed at low level.

Specifically, in any of the row exposure array and the uniform exposurearray, in order to select a pixel having the same color and exposuretime as the pixel of interest and being in the vertical direction, it isnecessary to store a pixel area corresponding to at least nine rows in amemory or the like.

FIGS. 16A and 16B are each a diagram for describing a pixel area that isnecessary to select pixels having the same color and exposure time asthe pixel of interest and being in the vertical direction. FIG. 16Aindicates an example of the row exposure array, and FIG. 16B indicatesan example of the uniform exposure array. In any of FIGS. 16A and 16B,in order to select pixels having the same color and exposure time as thepixel of interest and being in the vertical direction, it is necessaryto prepare a pixel area corresponding to nine rows.

In contrast to this, in the present disclosure, as described above withreference to FIG. 14, the pixel area corresponding to five rows allowsthe correction of a defective pixel to be performed.

Next, the example of the defect correction processing in the rowexposure array by the defect correction processing unit 21 will bedescribed with reference to the flowchart of FIG. 17.

In Step S21, the gain correcting unit 41 performs correction foradjusting a difference caused by the difference in exposure time on thepixel values of the input image. For example, the gain correcting unit41 multiplies the pixel value of the short-time exposure pixel by a gainthat is calculated as (exposure time of long-time exposurepixel/exposure time of short-time exposure pixel). This allows the pixelvalue of the short-time exposure pixel to be corrected to a pixel valuecorresponding to the long-time exposure.

It should be noted that in the processing subsequent to Step S21, notonly a pixel value after the correction by the gain correcting unit 41but also a pixel value before the correction can be referred to asappropriate.

In Step S22, the saturation determining unit 42 determines whether thepixel values are saturated or not for a processing unit area that is anarea having a predetermined number of pixels centering on the pixel ofinterest. Specifically, the saturation determining unit 42 determineswhether the processing unit area corresponds to a significantly brightarea in the image or not.

At that time, for example, in the case where the number of pixels whosepixel values are the maximum values is equal to or larger than a presetthreshold value, it is determined that the processing unit area issaturated. For example, in the case where the number of pixels whosepixel values are the maximum values is 3 (threshold value) or more, itis determined that the processing unit area is saturated.

In Step S22, in the case where the saturation determining unit 42determines that the processing unit area is not saturated, theprocessing proceeds to Step S23.

In Step S23, the flatness determining unit 43 executes flatnessdetermination processing in an unsaturated state that will be describedlater with reference to FIG. 18. Through this processing, it isdetermined whether the processing unit area is flat or not.

Here, a detailed example of the flatness determination processing in theunsaturated state in Step S23 of FIG. 17 will be described withreference to the flowchart of FIG. 18.

In Step S41, the flatness determining unit 43 compares the pixel valueof the pixel of interest with the maximum or minimum value in the pixelvalues of pixels having the same color as the pixel of interest in theprocessing unit area.

In Step S42, as a result of the comparison in Step S41, the flatnessdetermining unit 43 determines whether the pixel value of the pixel ofinterest is larger than the maximum value or not. In Step S42, in thecase where the flatness determining unit 43 determines that the pixelvalue of the pixel of interest is larger than the maximum value, theprocessing proceeds to Step S43.

In Step S43, the pixel of interest is regarded as a candidate of thewhite-dot defective pixel and a predetermined flag or the like is settherefor.

On the other hand, in Step S42, in the case where the flatnessdetermining unit 43 determines that the pixel value of the pixel ofinterest is not larger than the maximum value, the processing proceedsto Step S44.

In Step S44, as a result of the comparison in Step S41, the flatnessdetermining unit 43 determines whether the pixel value of the pixel ofinterest is smaller than the minimum value or not. In Step S44, in thecase where the flatness determining unit 43 determines that the pixelvalue of the pixel of interest is smaller than the minimum value, theprocessing proceeds to Step S45.

In Step S45, the pixel of interest is regarded as a candidate of theblack-dot defective pixel and a predetermined flag or the like is settherefor.

In Step S44, in the case where the flatness determining unit 43determines that the pixel value of the pixel of interest is not smallerthan the minimum value, the processing of Step S45 is skipped.

In Step S46, the flatness determining unit 43 calculates an averagevalue “ave_area” of the pixel values of the pixels having the same coloras the pixel of interest. At that time, the average value “ave_area” iscalculated by the expression (1), for example.

In Step S47, the flatness determining unit 43 calculates an averagedeviation “std_area” of the processing unit area by using the result ofthe calculation in Step S46. At that time, the average deviation“std_area” is calculated by the expression (2), for example.

In Step S48, the flatness determining unit 43 determines whether theaverage deviation “std_area” calculated in the processing of Step S47 isequal to or larger than a preset threshold value.

In Step S48, in the case where the flatness determining unit 43determines that the average deviation “std_area” is smaller than thethreshold value, the processing proceeds to Step S49 and the flatnessdetermining unit 43 determines that the processing unit area is flat.

In Step S48, in the case where the flatness determining unit 43determines that the average deviation “std_area” is equal to or largerthan the threshold value, the processing proceeds to Step S50 and theflatness determining unit 43 determines that the processing unit area isnot flat.

In such a manner, the flatness determination processing in theunsaturated state is executed.

Referring back to FIG. 17, in Step S24, the flatness determining unit 43determines whether the result of the determination in Step S23 is “flat”or not. In the case where the result of the determination in Step S23 is“flat”, the processing proceeds to Step S25.

In Step S25, the defect determining unit 45 executes defectdetermination processing in a flat state that will be described laterwith reference to FIG. 19. Through this processing, it is determinedwhether the pixel of interest is a defective pixel or not.

In Step S26, the defect correcting unit 46 executes defect correctionprocessing in a flat state that will be described later with referenceto FIG. 20. Through this processing, the pixel value of the pixel ofinterest as a defective pixel is corrected.

Here, a detailed example of the defect determination processing in theflat state in Step S25 of FIG. 17 will be described with reference tothe flowchart of FIG. 19.

In Step S61, the defect determining unit 45 calculates a threshold valuefor defect determination. At that time, the threshold value iscalculated by the expression (3), for example.

In Step S62, the defect determining unit 45 determines whether the pixelof interest is a candidate of the white-dot defective pixel or not. Inthe case where the defect determining unit 45 determines that the pixelof interest is a candidate of the white-dot defective pixel, theprocessing proceeds to Step S63.

In Step S63, the defect determining unit 45 determines whether adifference between the pixel value of the pixel of interest and themaximum value in the pixel values of pixels having the same color as thepixel of interest in the processing unit area is equal to or larger thanthe threshold value calculated in Step S61. For example, the defectdetermining unit 45 determines whether the absolute value of adifference between the maximum value in the pixel values of the R₀ to R₇of FIG. 6 and the pixel value of the pixel of interest is equal to orlarger than the threshold value described above.

In Step S63, in the case where the defect determining unit 45 determinesthat the difference is equal to or larger than the threshold value, theprocessing proceeds to Step S64, and it is determined that the pixel ofinterest is a white-dot defective pixel. In Step S63, in the case wherethe defect determining unit 45 determines that the difference is notequal to or larger than the threshold value, it is determined that thepixel of interest is not a defective pixel.

In Step S62, in the case where the defect determining unit 45 determinesthat the pixel of interest is not a candidate of the white-dot defectivepixel, the processing proceeds to Step S65.

In Step S65, the defect determining unit 45 determines whether the pixelof interest is a candidate of the black-dot defective pixel or not. Inthe case where the defect determining unit 45 determines that the pixelof interest is a candidate of the black-dot defective pixel, theprocessing proceeds to Step S66. In Step S65, in the case where thedefect determining unit 45 determines that the pixel of interest is nota candidate of the black-dot defective pixel, it is determined that thepixel of interest is not a defective pixel.

In Step S66, the defect determining unit 45 determines whether adifference between the minimum value in the pixel values of the pixelshaving the same color as the pixel of interest in the processing unitarea and the pixel value of the pixel of interest is equal to or largerthan the threshold value calculated in Step S61. For example, the defectdetermining unit 45 determines whether the absolute value of adifference between the minimum value in the pixel values of the R₀ to R₇of FIG. 6 and the pixel value of the pixel of interest is equal to orlarger than the threshold value described above.

In Step S66, in the case where the defect determining unit 45 determinesthat the difference is equal to or larger than the threshold value, theprocessing proceeds to Step S67, and it is determined that the pixel ofinterest is a black-dot defective pixel. In Step S66, in the case wherethe defect determining unit 45 determines that the difference is notequal to or larger than the threshold value, it is determined that thepixel of interest is not a defective pixel.

In such a manner, the defect determination processing in the flat stateis executed.

Next, the detailed example of the defect correction processing in theflat state in Step S26 of FIG. 17 will be described with reference tothe flowchart of FIG. 20.

In Step S81, it is determined whether the pixel of interest is awhite-dot defective pixel or not. In the case where the pixel ofinterest is determined to be a white-dot defective pixel, the processingproceeds to Step S82.

In Step S82, the defect correcting unit 46 replaces the pixel value ofthe pixel of interest with a larger one of the pixel values of thepixels having the same color and exposure time as the pixel of interestin the processing unit area. At that time, for example, the pixel valueof the pixel of interest is replaced with a larger one of the pixelvalues of the R₃ and the R₄ in FIG. 6.

On the other hand, in Step S81, in the case where it is determined thatthe pixel of interest is not a white-dot defective pixel, the processingproceeds to Step S83.

In Step S83, it is determined whether the pixel of interest is ablack-dot defective pixel or not. In the case where the pixel ofinterest is determined to be a black-dot defective pixel, the processingproceeds to Step S84.

In Step S84, the defect correcting unit 46 replaces the pixel value ofthe pixel of interest with a smaller one of the pixel values of thepixels having the same color and exposure time as the pixel of interestin the processing unit area. At that time, for example, the pixel valueof the pixel of interest is replaced with a smaller one of the pixelvalues of the R₃ and the R₄ in FIG. 6.

In such a manner, the defect correction processing in the flat state isexecuted.

Referring back to FIG. 17, in Step S24, in the case where the flatnessdetermining unit 43 determines that the processing unit area is notflat, the processing proceeds to Step S27.

In Step S27, the direction detecting unit 44 executes directiondetection processing that will be described later with reference to FIG.21. Through this processing, the direction of the texture is detected.

In Step S28, the defect determining unit 45 executes defectdetermination processing in a non-flat state that will be describedlater with reference to FIG. 22. Through this processing, it isdetermined whether the pixel of interest is a defective pixel or not.

In Step S29, the defect correcting unit 46 executes defect correctionprocessing in a non-flat state that will be described later withreference to FIG. 23. Through this processing, the pixel value of thepixel of interest as a defective pixel is corrected.

Here, a detailed example of the direction detection processing of StepS27 in FIG. 17 will be described with reference to the flowchart of FIG.21.

In Step S101, the direction detecting unit 44 calculates absolute valuesof differences between pixels having the same color and being adjacentin the horizontal direction. At that time, the direction detecting unit44 extracts a direction detection area that is formed of 35 (=7×5)pixels and is obtained by expanding the right and left ends of theprocessing unit area by one pixel. Subsequently, for example, asindicated by the arrows of the upper left part of FIG. 7, the absolutevalues of the differences between 15 sets of pixels, each set having thesame color, are calculated.

In Step S102, the direction detecting unit 44 calculates absolute valuesof differences between pixels having the same color and being adjacentin the vertical direction. At that time, for example, as indicated bythe arrows of the upper right part of FIG. 7, the absolute values of thedifferences between 15 sets of pixels, each set having the same color,are calculated.

In Step S103, the direction detecting unit 44 calculates absolute valuesof the differences between pixels having the same color and beingadjacent in the +45° direction. At that time, for example, as indicatedby the arrows of the lower left part of FIG. 7, the absolute values ofthe differences between 15 sets of pixels, each set having the samecolor, are calculated.

In Step S104, the direction detecting unit 44 calculates absolute valuesof the differences between pixels having the same color and beingadjacent in the −45° direction. At that time, for example, as indicatedby the arrows of the lower right part of FIG. 7, the absolute values ofthe differences between 15 sets of pixels, each set having the samecolor, are calculated.

In Step S105, the direction detecting unit 44 calculates an averagevalue of the absolute values of the differences in each of thedirections. At that time, as described above, in order to eliminate theinfluence of the defective pixel in the area, the direction detectingunit 44 selects 11 low-order absolute values of the differences from the15 absolute values of the differences in each of the directions.Subsequently, the direction detecting unit 44 calculates an averagevalue of the 11 absolute values of the differences in each of thedirections. It should be noted that the average value of the 11 absolutevalues of the differences in the horizontal direction and the averagevalue of the 11 absolute values of the differences in the verticaldirection are each multiplied by A/2.

In Step S106, the direction detecting unit 44 selects the minimum valueof the 4 average values corresponding to the respective directions anddetects a direction corresponding to the minimum value as the directionof the texture.

In such a manner, the direction detection processing is executed.

Next, a detailed example of the defect determination processing in thenon-flat state in Step S28 of FIG. 17 will be described with referenceto the flowchart of FIG. 22.

In Step S121, the defect determining unit 45 calculates the thresholdvalue for the defect determination. At that time, for example, thethreshold value is calculated by the expression (4).

In Step S122, the defect determining unit 45 determines whether adifference between the pixel value of the pixel of interest and theaverage value of the pixel values of pixels (for example, the R₀ to theR₇ in FIG. 6) having the same color in the processing unit area is equalto or larger than the threshold value described above. In Step S122, inthe case where the defect determining unit 45 determines that thedifference is not equal to or larger than the threshold value, it isdetermined that the pixel of interest is not a defective pixel. In StepS122, in the case where the defect determining unit 45 determines thatthe difference is equal to or larger than the threshold value, theprocessing proceeds to Step S123.

In Step S123, the defect determining unit 45 determines whether thepixel value of the pixel of interest is larger than the average value ofthe pixel values of the pixels having the same color in the processingunit area.

In Step S123, in the case where the defect determining unit 45determines that the pixel value of the pixel of interest is larger thanthe average value of the pixel values of the pixels having the samecolor in the processing unit area, the processing proceeds to Step S124and it is determined that the pixel of interest is a white-dot defectivepixel.

In Step S123, in the case where the defect determining unit 45determines that the pixel value of the pixel of interest is not larger(is smaller) than the average value of the pixel values of the pixelshaving the same color in the processing unit area, the processingproceeds to Step S125 and it is determined that the pixel of interest isa black-dot defective pixel.

In such a manner, the defect determination processing in the non-flatstate is executed.

Next, a detailed example of the defect correction processing in thenon-flat state in Step S29 of FIG. 17 will be described with referenceto the flowchart of FIG. 23.

In Step S141, the defect correcting unit 46 determines whether the pixelof interest is a black-dot defective pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is ablack-dot defective pixel, the processing proceeds to Step S142.

In Step S142, the defect correcting unit 46 selects pixels to be usedfor the replacement of pixel values based on the detected direction ofthe texture as a result of the processing of Step S27.

In Step S143, the defect correcting unit 46 performs correction byreplacing the pixel value of the pixel of interest with a smaller one ofthe pixel values of the pixels selected by the processing of Step S142.

On the other hand, in Step S141, in the case where the defect correctingunit 46 determines that the pixel of interest is not a black-dotdefective pixel, the processing proceeds to Step S144.

In Step S144, the defect correcting unit 46 determines whether the pixelof interest is a white-dot defective pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is awhite-dot defective pixel, the processing proceeds to Step S145.

In Step S145, the defect correcting unit 46 selects pixels to be usedfor the replacement of pixel values based on the detected direction ofthe texture as a result of the processing of Step S27.

In Step S146, the defect correcting unit 46 performs correction byreplacing the pixel value of the pixel of interest with a larger one ofthe pixel values of the pixels selected by the processing of Step S145.

For example, in the case where the direction of the texture is thehorizontal direction, when it is determined that the pixel of interestis a white-dot defective pixel, the pixel value of the pixel of interestis replaced with a larger one of the pixel values of the R₃ and the R₄of FIG. 6, and when it is determined that the pixel of interest is ablack-dot defective pixel, the pixel value of the pixel of interest isreplaced with a smaller one of the pixel values of the R₃ and the R₄ ofFIG. 6. Further, in the case where the direction of the texture is thevertical direction, when it is determined that the pixel of interest isa white-dot defective pixel, the pixel value of the pixel of interest isreplaced with a larger one of the pixel values of the R₁ and the R₆ ofFIG. 6, and when it is determined that the pixel of interest is ablack-dot defective pixel, the pixel value of the pixel of interest isreplaced with a smaller one of the pixel values of the R₁ and the R₆ ofFIG. 6. Additionally, in the case where the direction of the texture isthe +45° direction, when it is determined that the pixel of interest isa white-dot defective pixel, the pixel value of the pixel of interest isreplaced with a larger one of the pixel values of the R₂ and the R₅ ofFIG. 6, and when it is determined that the pixel of interest is ablack-dot defective pixel, the pixel value of the pixel of interest isreplaced with a smaller one of the pixel values of the R₂ and the R₅ ofFIG. 6. Additionally, in the case where the direction of the texture isthe −45° direction, when it is determined that the pixel of interest isa white-dot defective pixel, the pixel value of the pixel of interest isreplaced with a larger one of the pixel values of the R₀ and the R₇ ofFIG. 6, and when it is determined that the pixel of interest is ablack-dot defective pixel, the pixel value of the pixel of interest isreplaced with a smaller one of the pixel values of the R₀ and the R₇ ofFIG. 6.

In Step S147, the defect correcting unit 46 determines whether the pixelof interest is a short-time exposure pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is ashort-time exposure pixel, the processing proceeds to Step S148.

In Step S148, the defect correcting unit 46 multiplies the pixel value,which is replaced in Step S143 or S146, by a gain that is calculated as(exposure time of short-time exposure pixel/exposure time of long-timeexposure pixel), and thus eliminates the influence of the correction bythe gain correcting unit 41 (that is, inversely corrects the pixelvalue).

In such a manner, the defect correction processing in the non-flat stateis executed.

Referring back to FIG. 17, in Step S22, in the case where it isdetermined that the processing unit area is saturated, the processingproceeds to Step S30.

In Step S30, the defect determining unit 45 executes defectdetermination processing in a saturated state that will be describedlater with reference to FIG. 24. Through this processing, it isdetermined whether the pixel of interest is a defective pixel or not.

In Step S31, the defect correcting unit 46 executes defect correctionprocessing in a saturated state that will be described later withreference to FIG. 25. Through this processing, the pixel value of thepixel of interest as a defective pixel is corrected.

Here, a detailed example of the defect determination processing in thesaturated state of Step S30 of FIG. 17 will be described with referenceto the flowchart of FIG. 24.

In Step S161, the defect determining unit 45 calculates a dynamic rangeof 9 pixels that are arranged in the horizontal direction whilecentering on the pixel of interest.

At that time, for example, an area formed of 9 (1×9) pixels shown inFIG. 8 is extracted as a dynamic range detection area. It should benoted that when the saturation determining unit 42 determines that theprocessing unit area is saturated, a calculation using a pixel valuebefore the correction by the gain correcting unit 41 is performed.Subsequently, for pixels having the same color as the pixel of interestand being indicated by the circles in FIG. 8 (in this example, Rpixels), the defect determining unit 45 calculates 4 absolute values ofdifferences between the pixel values of those pixels that are adjacentas indicated by the arrows in FIG. 8, and calculates a dynamic range inthe 4 absolute values of the differences. Further, the defectdetermining unit 45 calculates a dynamic range of the pixel values ofpixels having a color that is different from the color of the pixel ofinterest (in this example, G pixels).

In Step S162, the defect determining unit 45 calculates a thresholdvalue for the defect determination. At that time, for example, thedefect determining unit 45 calculates a square root of the average valueof the pixel values of the pixels having the same color as the pixel ofinterest and being indicated by the circles in FIG. 8, and multiplies bya parameter or adding a parameter as appropriate, to set the resultantvalue as a threshold value for defect determination. Since first tothird conditions are used for the threshold value as described later,three threshold values are calculated with changed parameters.

In Step S163, the defect determining unit 45 determines whether all ofthree conditions on white-dot defect are satisfied or not.

Specifically, the first condition is as follows: the pixel value of thepixel of interest is larger than the second-largest pixel value in thepixel values of the 4 pixels having the same color and being indicatedby the circles in FIG. 8, and a difference between the pixel value ofthe pixel of interest and the second-largest pixel value in the pixelvalues of the 4 pixels indicated by the circles in FIG. 8 is larger thanthe threshold value.

In consideration of the case where defective pixels having the samecolor alternately exist, like the order of a defective pixel, anon-defective pixel, a defective pixel, . . . , for example, adifference between the pixel value of the pixel of interest and thelargest pixel value is not compared with the threshold value, and thedifference between the pixel value of the pixel of interest and thesecond-largest pixel value is compared with the threshold value.

The second condition is as follows: the dynamic range in the absolutevalues of the differences between the pixel values of the adjacentpixels having the same color as the pixel of interest and beingindicated by the circles in FIG. 8 is larger than the threshold value.

The third condition is as follows: the dynamic range of the pixel valuesof pixels having a color that is different from the color of the pixelof interest shown in FIG. 8 (in FIG. 8, G pixels) is smaller than thethreshold value.

In Step S163, in the case where all of the first to third conditionsdescribed above are satisfied, the processing proceeds to Step S164 andit is determined that the pixel of interest is a white-dot defectivepixel.

In Step S163, in the case where at least one of the three conditions onthe white-dot defect is not satisfied, the processing proceeds to StepS165.

In Step S165, the defect determining unit 45 determines whether all ofthree conditions on black-dot defect are satisfied or not.

Specifically, the first condition is as follows: the pixel value of thepixel of interest is smaller than the second-smallest pixel value in thepixel values of the 4 pixels having the same color and being indicatedby the circles in FIG. 8, and a difference between the pixel value ofthe pixel of interest and the second-smallest pixel value in the pixelvalues of the 4 pixels indicated by the circles in FIG. 8 is larger thanthe threshold value.

The second and third conditions are the same as in the case of thewhite-dot defective pixel, and thus their detailed description will beomitted.

In Step S165, in the case where all of the first to third conditionsdescribed above are satisfied, the processing proceeds to Step S166 andit is determined that the pixel of interest is a black-dot defectivepixel.

In Step S165, in the case where at least one of the three conditions onthe black-dot defect is not satisfied, this means that the pixel ofinterest is determined not to be a defective pixel.

In such a manner, the defect determination processing in the saturatedstate is executed.

Next, a detailed example of the defect correction processing in thesaturated state of Step S31 of FIG. 17 will be described with referenceto the flowchart of FIG. 25.

In Step S181, the defect correcting unit 46 determines whether the pixelof interest is a black-dot defective pixel or not as a result of theprocessing of Step S30.

In Step S181, in the case where the defect correcting unit 46 determinesthat the pixel of interest is a black-dot defective pixel, theprocessing proceeds to Step S182.

In Step S182, the defect correcting unit 46 replaces the pixel value ofthe pixel of interest with the second-smallest pixel value in the pixelvalues of the pixels having the same color as the pixel of interest andbeing indicated by the circles in FIG. 8.

On the other hand, in Step S181, in the case where the defect correctingunit 46 determines that the pixel of interest is not a black-dotdefective pixel, the processing proceeds to Step S183.

In Step S183, the defect correcting unit 46 determines whether the pixelof interest is a white-dot defective pixel or not as a result of theprocessing of Step S30. In Step S183, in the case where the defectcorrecting unit 46 determines that the pixel of interest is a white-dotdefective pixel, the processing proceeds to Step S184.

In Step S184, the defect correcting unit 46 replaces the pixel value ofthe pixel of interest with the second-largest pixel value in the pixelvalues of the pixels having the same color as the pixel of interest andbeing indicated by the circles in FIG. 8.

In such a manner, the defect correction processing in the saturatedstate is executed.

As described above, the defect correction processing in the row exposurearray is executed.

Next, the example of the defect correction processing in the uniformexposure array by the defect correction processing unit 21 will bedescribed with reference to the flowchart of FIG. 26.

In Step S201, the gain correcting unit 41 performs correction foradjusting a difference caused by the difference in exposure time on thepixel values of the input image. For example, the gain correcting unit41 multiplies the pixel value of the short-time exposure pixel by a gainthat is calculated as (exposure time of long-time exposurepixel/exposure time of short-time exposure pixel). This allows the pixelvalue of the short-time exposure pixel to be corrected to a pixel valuecorresponding to the long-time exposure.

It should be noted that in the processing subsequent to Step S201, notonly a pixel value after the correction by the gain correcting unit 41but also a pixel value before the correction can be referred to asappropriate.

In Step S202, the saturation determining unit 42 determines whether thepixel values are saturated or not for a processing unit area that is anarea having a predetermined number of pixels centering on the pixel ofinterest. Specifically, the saturation determining unit 42 determineswhether the processing unit area corresponds to a significantly brightarea in the image or not.

At that time, for example, in the case where the number of pixels whosepixel values are the maximum values is equal to or larger than a presetthreshold value, it is determined that the processing unit area issaturated. For example, in the case where the number of pixels whosepixel values are the maximum values is 3 (threshold value) or more, itis determined that the processing unit area is saturated.

In Step S202, in the case where the saturation determining unit 42determines that the processing unit area is not saturated, theprocessing proceeds to Step S203.

In Step S203, the flatness determining unit 43 executes flatnessdetermination processing in an unsaturated state that will be describedlater with reference to FIG. 18. Through this processing, it isdetermined whether the processing unit area is flat or not.

The processing of Step S203 of FIG. 26 is the same as the processing ofStep S23 of FIG. 17, and thus its detailed description will be omitted.However, in Step S203, when the pixel of interest is a G pixel, it isassumed that pixels having the same color as the pixel of interest aredisposed in the processing unit area as shown in FIG. 11. Specifically,in the processing unit area, all the pixels having the same color as thepixel of interest have the same exposure time (in this example,short-time exposure pixels).

Consequently, in Step S203, when the pixel of interest is a G pixel, acalculation using a pixel value before correction by the gain correctingunit 41 is performed. Further, three threshold values, that is, athreshold value for an R or B pixel, a threshold value for a G long-timeexposure pixel, and a threshold value for a G short-time exposure pixel,are prepared as threshold values used for determining whether theprocessing unit area is flat or not.

In Step S204, the flatness determining unit 43 determines whether theresult of the determination in Step S203 is “flat” or not. In the casewhere the result of the determination in Step S23 is “flat”, theprocessing proceeds to Step S205.

In Step S205, the defect determining unit 45 executes defectdetermination processing in a flat state. Through this processing, it isdetermined whether the pixel of interest is a defective pixel or not.

The processing of Step S205 is the same as the processing of Step S25 ofFIG. 17, and thus its detailed description will be omitted. In StepS205, however, when a threshold value for defect determination iscalculated, the defect determining unit 45 calculates three thresholdvalues, that is, a threshold value for an R or B pixel, a thresholdvalue for a G long-time exposure pixel, and a threshold value for a Gshort-time exposure pixel.

In Step S206, the defect correcting unit 46 executes defect correctionprocessing in a flat state. Through this processing, the pixel value ofthe pixel of interest as a defective pixel is corrected.

The processing of Step S206 is the same as the processing of Step S26 ofFIG. 17, and thus its detailed description will be omitted. In StepS206, however, when the pixel of interest is an R pixel, the R₀, R₂, R₅,or R₇ in FIG. 10 is used as a pixel to be replaced, and when the pixelof interest is a G pixel, the G₀ to G₇ in FIG. 11 are used as a pixel tobe replaced. It should be noted that when the pixel of interest is a Bpixel, as in the case where the pixel of interest is an R pixel, a pixelhaving the same color and exposure time as the pixel of interest in theprocessing unit area is used.

On the other hand, in Step S204, in the case where the flatnessdetermining unit 43 determines that the processing unit area is notflat, the processing proceeds to Step S207.

In Step S207, the direction detecting unit 44 executes directiondetection processing. Through this processing, the direction of thetexture is detected.

In Step S208, the defect determining unit 45 executes defectdetermination processing in a non-flat state. Through this processing,it is determined whether the pixel of interest is a defective pixel ornot.

In Step S209, the defect correcting unit 46 executes defect correctionprocessing in a non-flat state. Through this processing, the pixel valueof the pixel of interest as a defective pixel is corrected.

The processing from Step S207 to Step S209 is the same as the processingfrom Step S27 to Step S29 of FIG. 17, and thus their detaileddescription will be omitted.

On the other hand, in Step S202, in the case where the saturationdetermining unit 42 determines that the processing unit area issaturated, the processing proceeds to Step S210.

In Step S210, the flatness determining unit 43 executes flatnessdetermination processing in a saturated state. Through this processing,it is determined whether the processing unit area is flat or not.

Here, a detailed example of the flatness determination processing in thesaturated state of Step S210 of FIG. 26 will be described with referenceto the flowchart of FIG. 27.

In Step S221, the flatness determining unit 43 determines whether thepixel of interest is a G pixel or not. In the case where the flatnessdetermining unit 43 determines that the pixel of interest is a G pixel,the processing proceeds to Step S222.

In Step S222, the flatness determining unit 43 compares the pixel valueof the pixel of interest with the maximum or minimum value in the pixelvalues of pixels having the same color as the pixel of interest in theprocessing unit area.

On the other hand, in Step S221, in the case where the flatnessdetermining unit 43 determines that the pixel of interest is not a Gpixel (specifically, the pixel of interest is an R or B pixel), theprocessing proceeds to Step S223.

In Step S223, the flatness determining unit 43 compares the pixel valueof the pixel of interest with the maximum or minimum value in the pixelvalues of 4 pixels having the same color and exposure time as the pixelof interest and being indicated by the circles in FIG. 12, for example.

In Step S224, as a result of the comparison in Step S222 or Step S223,the flatness determining unit 43 determines whether the pixel value ofthe pixel of interest is larger than the maximum value or not. In StepS224, in the case where the flatness determining unit 43 determines thatthe pixel value of the pixel of interest is larger than the maximumvalue, the processing proceeds to Step S225.

In Step S225, the pixel of interest is regarded as a candidate of thewhite-dot defective pixel and a predetermined flag or the like is settherefor.

On the other hand, in Step S224, as a result of the comparison in StepS222 or Step S223, in the case where the flatness determining unit 43determines that the pixel value of the pixel of interest is not largerthan the maximum value, the processing proceeds to Step S226.

In Step S226, as a result of the comparison in Step S222 or Step S223,the flatness determining unit 43 determines whether the pixel value ofthe pixel of interest is smaller than the minimum value or not. In StepS226, in the case where the flatness determining unit 43 determines thatthe pixel value of the pixel of interest is smaller than the minimumvalue, the processing proceeds to Step S227.

In Step S227, the pixel of interest is regarded as a candidate of theblack-dot defective pixel and a predetermined flag or the like is settherefor.

In Step S226, in the case where the flatness determining unit 43determines that the pixel value of the pixel of interest is not smallerthan the minimum value, the processing of Step S227 is skipped.

In Step S228, the flatness determining unit 43 calculates an averagevalue “ave_area” of the pixel values of the pixels having the same coloras the pixel of interest. At that time, the average value “ave_area” iscalculated by the expression (5), for example.

In Step S229, the flatness determining unit 43 calculates an averagedeviation “std_area” of the processing unit area by using the result ofthe calculation in Step S228. At that time, the average deviation“std_area” is calculated by the expression (6), for example.

In Step S230, the flatness determining unit 43 determines whether theaverage deviation “std_area” calculated in the processing of Step S229is equal to or larger than a preset threshold value.

It should be noted that in Step S230, 4 threshold values, that is, athreshold value for an R or B long-time exposure pixel, a thresholdvalue for an R or B short-time exposure pixel, a threshold value for a Glong-time exposure pixel, and a threshold value for a G short-timeexposure pixel, are prepared as threshold values used for determiningwhether the processing unit area is flat or not.

In Step S230, in the case where the flatness determining unit 43determines that the average deviation “std_area” is smaller than thethreshold value, the processing proceeds to Step S231 and the flatnessdetermining unit 43 determines that the processing unit area is flat.

In Step S230, in the case where the flatness determining unit 43determines that the average deviation “std_area” is equal to or largerthan the threshold value, the processing proceeds to Step S232 and theflatness determining unit 43 determines that the processing unit area isnot flat.

In such a manner, the flatness determination processing in the saturatedstate is executed.

Referring back to FIG. 26, in Step S211, the flatness determining unit43 determines whether the processing unit area is flat or not as aresult of the processing of Step S210. In the case where the processingunit area is determined to be flat, the processing proceeds to StepS212.

In Step S212, the defect determining unit 45 executes defectdetermination processing in a saturated and flat state. Through thisprocessing, it is determined whether the pixel of interest is adefective pixel or not.

The processing of Step S212 is the same as the processing of Step S25 ofFIG. 17, and thus its detailed description will be omitted. In StepS212, however, when a threshold value for defect determination iscalculated, the defect determining unit 45 calculates three thresholdvalues, that is, a threshold value for an R or B pixel, a thresholdvalue for a G long-time exposure pixel, and a threshold value for a Gshort-time exposure pixel.

In Step S213, the defect correcting unit 46 executes defect correctionprocessing in a saturated and flat state. Through this processing, thepixel value of the pixel of interest as a defective pixel is corrected.

The processing of Step S213 is the same as the processing of Step S26 ofFIG. 17, and thus its detailed description will be omitted. In StepS206, however, when the pixel of interest is an R pixel, the R₀, R₂, R₅,or R₇ in FIG. 10 is used as a pixel to be replaced, and when the pixelof interest is a G pixel, the G₀ to G₇ in FIG. 11 are used as a pixel tobe replaced. It should be noted that when the pixel of interest is a Bpixel, as in the case where the pixel of interest is an R pixel, a pixelhaving the same color and exposure time as the pixel of interest in theprocessing unit area is used.

On the other hand, in Step S211, in the case where the flatnessdetermining unit 43 determines that the processing unit area is not flatas a result of the processing of Step S210, the processing proceeds toStep S214.

In Step S214, the direction detecting unit 44 executes directiondetection processing in a saturated state. Through this processing, thedirection of the texture is detected.

The processing of Step S214 is the same as the processing of Step S27 inFIG. 17.

In Step S214, since it is thought that the long-time exposure pixels aresaturated, the direction detection using only G short-time exposurepixels is performed.

Specifically, the direction detecting unit 44 extracts a directiondetection area that is formed of 35 (=7×5) pixels and is obtained byexpanding the right and left ends of the processing unit area by onepixel. The direction detecting unit 44 calculates absolute values ofdifferences between adjacent G pixels in the direction detection area.At that time, as shown in FIG. 13, the direction detecting unit 44calculates absolute values of differences between adjacent G pixels ineach of 4 directions, that is, a horizontal direction, a verticaldirection, a +45° direction, and −45° direction.

Subsequently, in order to eliminate the influence of the defective pixelin the area, the direction detecting unit 44 selects 4 low-orderabsolute values of differences from the absolute values of thedifferences in each of the directions and calculates an average value ofthe 4 absolute values of the differences in each of the directions. Itshould be noted that the average value of the 4 absolute values of thedifferences in the horizontal direction and the average value of the 4absolute values of the differences in the vertical direction are eachmultiplied by A/2.

Additionally, the direction detecting unit 44 selects the minimum valueof the 4 average values corresponding to the respective directions anddetects a direction corresponding to the minimum value as the directionof the texture.

In the case where the pixel of interest is a G short-time exposure pixelor a B pixel, when an area formed of 35 (=7×5) pixels in which thenumber of G short-time exposure pixels becomes the largest is extracted,the position of the pixel of interest does not come to the center of thearea. In such a case, the result of the direction detection with a pixellocated on the left of the pixel of interest being at the center isused.

In Step S215, the defect determining unit 45 executes defectdetermination processing in a saturated and non-flat state that will bedescribed later with reference to FIG. 28. Through this processing, itis determined whether the pixel of interest is a defective pixel or not.

In Step S216, the defect correcting unit 46 executes defect correctionprocessing in a saturated and non-flat state that will be describedlater with reference to FIGS. 29 to 31. Through this processing, thepixel value of the pixel of interest as a defective pixel is corrected.

Here, a detailed example of the defect determination processing in thesaturated and non-flat state of Step S215 in FIG. 26 will be describedwith reference to the flowchart of FIG. 28.

In Step S241, the defect determining unit 45 calculates an average valueof the pixel values of pixels having the same color and exposure time asthe pixel of interest in accordance with the color of the pixel ofinterest.

For example, in the case where the pixel of interest is an R or B pixel,the defect determining unit 45 calculates an average value of the pixelvalues of 4 pixels, which are located at positions expressed in whiteand indicated by the circles in FIG. 10. In the case where the pixel ofinterest is a G pixel, the defect determining unit 45 calculates anaverage value of the pixel values of 8 pixels, which are located atpositions indicated by the circles in FIG. 11.

In Step S242, the defect determining unit 45 calculates a thresholdvalue for defect determination. At that time, the defect determiningunit 45 calculates a threshold value by using a square root of theaverage value calculated in Step S241 and using the average valuecorresponding to the direction detected by the direction detecting unit44 in the processing of Step S214 (the average value of the 4 absolutevalues of the differences). It should be noted that here, threethreshold values, that is, a threshold value for an R or B pixel, athreshold value for a G long-time exposure pixel, and a threshold valuefor a G short-time exposure pixel are calculated.

In Step S243, the defect determining unit 45 determines whether anabsolute value of a difference between the average value calculated inStep S241 and the pixel value of the pixel of interest is equal to orlarger than the threshold value calculated in Step S242. In Step S243,in the case where the defect determining unit 45 determines that theabsolute value of the difference is not equal to or larger than thethreshold value, it is determined that the pixel of interest is not adefective pixel.

In Step S243, in the case where the defect determining unit 45determines that the absolute value of the difference is equal to orlarger than the threshold value, the processing proceeds to Step S244.

In Step S244, the defect determining unit 45 determines whether thepixel value of the pixel of interest is larger than the average valuecalculated in Step S241 or not.

In Step S244, in the case where the defect determining unit 45determines that the pixel value of the pixel of interest is larger thanthe average value calculated in Step S241, the processing proceeds toStep S245 and it is determined that the pixel of interest is a white-dotdefective pixel.

On the other hand, in Step S244, in the case where the defectdetermining unit 45 determines that the pixel value of the pixel ofinterest is smaller than the average value calculated in Step S241, theprocessing proceeds to Step S246 and it is determined that the pixel ofinterest is a black-dot defective pixel.

In such a manner, the defect determination processing in the saturatedand non-flat state is executed.

Next, a detailed example of the defect correction processing in thesaturated and non-flat state in Step S216 of FIG. 26 will be describedwith reference to the flowcharts of FIGS. 29 to 31.

In Step S261, the defect correcting unit 46 determines whether the pixelof interest is a G pixel or not. In the case where the defect correctingunit 46 determines that the pixel of interest is a G pixel, theprocessing proceeds to Step S262.

In Step S262, the defect correcting unit 46 determines whether the pixelof interest is a black-dot defective pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is ablack-dot defective pixel, the processing proceeds to Step S263.

In Step S263, the defect correcting unit 46 selects pixels to be usedfor the replacement of pixel values based on the detected direction ofthe texture as a result of the processing of Step S214.

In Step S264, the defect correcting unit 46 performs correction byreplacing the pixel value of the pixel of interest with a smaller one ofthe pixel values of the pixels selected by the processing of Step S263.

On the other hand, in Step S262, in the case where the defect correctingunit 46 determines that the pixel of interest is not a black-dotdefective pixel, the processing proceeds to Step S265.

In Step S265, the defect correcting unit 46 determines whether the pixelof interest is a white-dot defective pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is awhite-dot defective pixel, the processing proceeds to Step S266.

In Step S266, the defect correcting unit 46 selects pixels to be usedfor the replacement of pixel values based on the detected direction ofthe texture as a result of the processing of Step S214.

In Step S267, the defect correcting unit 46 performs correction byreplacing the pixel value of the pixel of interest with a larger one ofthe pixel values of the pixels selected by the processing of Step S266.

In the processing of Steps S262 to S267, for example, in the case wherethe direction of the texture is the horizontal direction, the G₃ and theG₄ of FIG. 11 are selected. Further, in the case where the direction ofthe texture is the vertical direction, the G₁ and the G₆ of FIG. 11 areselected. Furthermore, in the case where the direction of the texture isthe +45° direction, the G₂ and the G₅ of FIG. 11 are selected.Additionally, in the case where the direction of the texture is the −45°direction, the G₀ and the G₇ of FIG. 11 are selected.

When it is determined that the pixel of interest is a white-dotdefective pixel, the pixel value of the pixel of interest is replacedwith a larger one of the pixel values of the two pixels selected asdescribed above. Further, when it is determined that the pixel ofinterest is a black-dot defective pixel, the pixel value of the pixel ofinterest is replaced with a smaller one of the pixel values of the twopixels selected as described above.

On the other hand, in Step S261, in the case where the defect correctingunit 46 determines that the pixel of interest is not a G pixel(specifically, the pixel of interest is an R or B pixel), the processingproceeds to Step S281 of FIG. 30.

In Step S281, the defect correcting unit 46 determines whether thedirection of the texture detected in the processing of Step S214 is thevertical direction or not. In the case where the defect correcting unit46 determines that the direction of the texture is not the verticaldirection, the processing proceeds to Step S282.

In Step S282, the defect correcting unit 46 determines whether the pixelof interest is a black-dot defective pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is ablack-dot defective pixel, the processing proceeds to Step S283.

In Step S283, the defect correcting unit 46 selects pixels having thesame color and exposure time as the pixel of interest based on thedirection of the texture detected in the processing of Step S214.

In Step S284, the defect correcting unit 46 replaces the pixel value ofthe pixel of interest with a smaller one of the pixel values of thepixels selected in the processing of Step S283.

On the other hand, in Step S282, in the case where the defect correctingunit 46 determines that the pixel of interest is not a black-dotdefective pixel, the processing proceeds to Step S285.

In Step S285, the defect correcting unit 46 determines whether the pixelof interest is a white-dot defective pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is awhite-dot defective pixel, the processing proceeds to Step S286.

In Step S286, the defect correcting unit 46 selects pixels having thesame color and exposure time as the pixel of interest based on thedirection of the texture detected in the processing of Step S214.

In Step S287, the defect correcting unit 46 replaces the pixel value ofthe pixel of interest with a larger one of the pixel values of thepixels selected in the processing of Step S286.

In the processing of Steps S282 to S287, for example, in the case wherethe direction of the texture is the +45° direction, the R₂ and the R₅ ofFIG. 14 are selected. In the case where the direction of the texture isthe −45° direction, the R₀ and the R₇ of FIG. 14 are selected.

In the case where the direction of the texture is the horizontaldirection, the R₃ and the R₄ of FIG. 14 are selected. In the uniformexposure array, a pixel that is closest to the pixel of interest in thehorizontal direction and has the same color as the pixel of interest isa pixel having exposure time that is different from the exposure time ofthe pixel of interest. So, it is necessary to select pixels from adirection detection area formed of 45 (=9×5) pixels, which is obtainedby further expanding the direction detection area formed of 35 (=7×5)pixels.

In the case where the pixel of interest is a black-dot defective pixel,the pixel value of the pixel of interest is replaced with a smaller oneof the pixel values of the selected pixels. In the case where the pixelof interest is a white-dot defective pixel, the pixel value of the pixelof interest is replaced with a larger one of the pixel values of theselected pixels.

On the other hand, in Step S281, in the case where the defect correctingunit 46 determines that the direction of the texture is the verticaldirection, the processing proceeds to Step S301 of FIG. 31.

In Step S301, the defect correcting unit 46 calculates pixel values ofvirtual pixels by linear interpolation.

Specifically, in the case where the direction of the texture is thevertical direction, the pixel values of pixels at two positions in thevertical direction are calculated by linear interpolation. For example,a virtual pixel value having a pixel value calculated as (R₀+R₂)/2 inFIG. 14 and a virtual pixel value having a pixel value calculated as(R₅+R₇)/2 are calculated. This is because, in the uniform exposurearray, pixels having the same color and exposure time do not exist inthe vertical direction in the area corresponding to five rows.

In Step S302, the defect correcting unit 46 determines whether the pixelof interest is a black-dot defective pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is ablack-dot defective pixel, the processing proceeds to Step S303.

In Step S303, the defect correcting unit 46 sets a smaller one of thevirtual pixel values calculated in the processing of Step S301 as acorrection candidate value.

On the other hand, in Step S302, in the case where the defect correctingunit 46 determines that the pixel of interest is not a black-dotdefective pixel, the processing proceeds to Step S304.

In Step S304, the defect correcting unit 46 determines whether the pixelof interest is a white-dot defective pixel or not. In the case where thedefect correcting unit 46 determines that the pixel of interest is awhite-dot defective pixel, the processing proceeds to Step S305.

In Step S305, the defect correcting unit 46 sets a larger one of thevirtual pixel values calculated in the processing of Step S301 as acorrection candidate value.

In Step S306, for the correction candidate value obtained in Step S303or S305, the pixel value of the short-time exposure pixel is correctedby being multiplied by a gain obtained by (exposure time of long-timeexposure pixel/exposure time of short-time exposure pixel).

In Step S307, the defect correcting unit 46 calculates an average valueof the pixel values of the pixels having the same color as the pixel ofinterest and being in the vertical direction. For example, in the caseof FIG. 14, (R₁+R₆)/2 is calculated. It should be noted that the R₁ andthe R₆ are the pixel values of the virtual pixels calculated by thelinear interpolation.

In Step S308, the defect correcting unit 46 calculates an absolute valueof a difference between the correction candidate value obtained in StepS303 or S305 and the average value obtained in Step S307.

In Step S309, the defect correcting unit 46 determines a mixing ratiowhen the pixel value of the pixel of interest is mixed to the correctioncandidate value, based on the absolute value of the difference obtainedin Step S308.

At that time, the mixing ratio is determined as described above withreference to FIG. 15, for example.

In Step S310, the defect correcting unit 46 mixes the pixel value of thepixel of interest to the correction candidate value, based on the mixingratio obtained in Step S309, to correct the pixel value of the pixel ofinterest.

In such a manner, the defect correction processing in the saturated andnon-flat state is executed.

As described above, the defect correction processing in the uniformexposure array is executed.

In the above description, the example in which the embodiment of thepresent disclosure is applied to an image sensor that obtains an imagein a high dynamic range by changing an exposure time in accordance withthe position of a pixel in the pixel array of the Bayer array, the imagein the high dynamic range being appropriately displayed from the pixelsof low luminance to the pixels of high luminance. However, theembodiment of the present disclosure is also applicable to an imagesensor that obtains an image in a high dynamic range by changing a lightsensitivity in accordance with the position of a pixel in the pixelarray of the Bayer array, the image in the high dynamic range beingappropriately displayed from the pixels of low luminance to the pixelsof high luminance.

Further, the image sensor to which the embodiment of the presentdisclosure is applied is not necessarily limited to the image sensorhaving the pixel array of the Bayer array.

FIG. 32 is a diagram showing a configuration example of a solid-stateimaging device serving as a semiconductor device according to theembodiment of the present disclosure. In FIG. 32, a solid-state imagingdevice 100 is an image sensor that adopts an SVE (Spatially VaryingExposure) method, for example, and is constituted as an image sensor 100that can execute the processing described above with reference to FIGS.17 to 31.

As shown in FIG. 32, the image sensor 100 has a laminate structureincluding a first chip (upper chip) 101 and a second chip (lower chip)102. The image sensor 100 is formed as a solid-state imaging devicehaving the laminate structure that is obtained after wafer bonding andcutting-out by singulation, as will be described later.

In the laminate structure of the two upper and lower chips, for example,the first chip 101 is constituted as an image sensor chip, and thesecond chip 102 is constituted as a logic chip including a controlcircuit and an image processing circuit for the first chip.

Bonding pads BPD and input and output circuits are formed in the secondchip (lower chip) 102. Openings OPN for wire bonding of the second chip102 are formed in the first chip (upper chip).

On the upper side of the image sensor 100 in FIG. 32 and at the centerof the first chip 101, a pixel array in which the short-time exposurepixels and the long-time exposure pixels are disposed in the rowexposure array or the uniform exposure array is formed.

In the second chip (lower chip) 102, the circuits for achieving thefunction of the image processing device 10 shown in FIG. 3, and the likeare formed.

FIG. 33 is a diagram for describing a process flow of the image sensorhaving the laminate structure according to this embodiment.

As shown in the left part of FIG. 33, wafers in which upper and lowerchips are manufactured by respective optimal processes are bonded toeach other. Subsequently, the back surface of the upper chip ispolished, and the wafer thickness of the upper chip is reduced. Afterpatterning is performed on the first chip (upper chip) 101,through-holes are formed to penetrate from the first chip 101 to awiring layer of the second chip (lower chip) 102 and embedded with metalto form vias.

As shown in the center part of FIG. 33, signal lines and power-supplylines between the upper and lower chips are electrically connectedthrough those vias.

As shown in the right part of FIG. 33, after processing of color filtersand microlenses is performed on the first chip (upper chip) 101, chipsare cut out by singulation.

FIG. 34 is a block diagram showing a configuration example of an imagingapparatus serving as an electronic apparatus to which the embodiment ofthe present disclosure is applied.

An imaging apparatus 600 of FIG. 34 includes an optical unit 601constituted of a lens group and the like, a solid-state imaging device(imaging device) 602, and a DSP (Digital Signal Processing) circuit 603serving as a camera signal processing circuit. Further, the imagingapparatus 600 includes a frame memory 604, a display 605, a recordingunit 606, an operation unit 607, and a power supply unit 608. The DSPcircuit 603, the frame memory 604, the display 605, the recording unit606, the operation unit 607, and the power supply unit 608 are connectedto one another via a bus line 609.

The optical unit 601 takes in incident light coming from a subject,i.e., an image light, and forms an image on an imaging surface of thesolid-state imaging device 602. The solid-state imaging device 602converts the amount of incident light, with which the image is formed onthe imaging surface by the optical unit 601, into an electrical signalon a pixel-by-pixel basis. The solid-state imaging device 602subsequently outputs the electrical signal as a pixel signal. Asolid-state imaging device such as the image sensor 100 according to theembodiment described above can be used as the solid-state imaging device602.

The display 605 is formed of a panel display such as a liquid crystalpanel or an organic EL (Electro Luminescence) panel and displays movingimages or still images captured by the solid-state imaging device 602.The recording unit 606 records the moving images or still imagescaptured by the solid-state imaging device 602 on a recording mediumsuch as a video tape and a DVD (Digital Versatile Disk).

The operation unit 607 issues operation commands on various functions ofthe imaging apparatus 600 under the operation of a user. The powersupply unit 608 supplies various types of power, which serves asoperation power of the DSP circuit 603, the frame memory 604, thedisplay 605, the recording unit 606, and the operation unit 607, tothose supply targets as appropriate.

Further, the embodiment of the present disclosure is not limited to beapplied to the solid-state imaging device that detects the distributionof the amount of incident visible light and captures the distribution asan image. The embodiment of the present disclosure is applicable to asolid-state imaging device that captures the distribution of theincident light amount of infrared rays or X rays or the distribution ofparticles or the like as an image, or applicable to, in a broader sense,all solid-state imaging devices (physical quantity distributiondetecting device) including a fingerprint detection sensor that detectsthe distribution of another physical quantity such as a pressure and anelectrostatic capacitance and captures an image of the distribution.

Furthermore, the embodiment of the present disclosure is not limited tothe embodiment described above and can be variously modified withoutdeparting from the gist of the present disclosure.

It should be noted that the present disclosure can have the followingconfigurations.

(1) A solid-state imaging device, including:

a pixel array including a plurality of pixels, the plurality of pixelseach having one of a different exposure time and a different exposuresensitivity and being disposed according to a predetermined rule; and

a pixel value correcting unit configured to correct, among pixel valuesobtained from the plurality of pixels in the pixel array, a pixel valueof a pixel of the plurality of pixels that applies to a presetcondition, by using a pixel value of another pixel of the plurality ofpixels.

(2) The solid-state imaging device according to (1), in which

the pixel array includes the plurality of pixels each having one of thesame exposure time and the same exposure sensitivity and being regularlydisposed on a row-by-row basis.

(3) The solid-state imaging device according to (1), in which

the pixel array includes the plurality of pixels including apredetermined number of pixels each having one of the same exposure timeand the same exposure sensitivity and being regularly disposed as anL-shaped group of pixels.

(4) The solid-state imaging device according to any one of (1) to (3),in which

the pixel value correcting unit is configured to set a pixel of interestin the plurality of pixels disposed in the pixel array to be the centerof the pixel array, extract a processing unit area including a presetnumber of rows of pixels, and correct the pixel value of the pixel ofinterest for each processing unit area.

(5) The solid-state imaging device according to (4), in which

the processing unit area includes five rows.

(6) The solid-state imaging device according to (4), in which

the pixel value correcting unit includes

-   -   a saturation determining unit configured to determine whether        the processing unit area is saturated or not based on the number        of pixels that output a maximum pixel value among the pixels of        the processing unit area,    -   a flatness determining unit configured to determine whether or        not an image formed of the pixels of the processing unit area is        a flat image that is free from a texture,    -   a direction detecting unit configured to detect a direction of        the texture when it is determined that the image formed of the        pixels of the processing unit area is not a flat image,    -   a defect determining unit configured to determine whether the        pixel of interest is a defective pixel or not, and    -   a defect correcting unit configured to correct the pixel value        of the pixel of interest when it is determined that the pixel of        interest is a defective pixel.

(7) The solid-state imaging device according to (6), in which

in accordance with a result of the determination by the saturationdetermining unit, the flatness determining unit is configured todetermine whether the image is a flat image or not, and the directiondetecting unit is configured to detect the direction of the texture, bydifferent methods.

(8) The solid-state imaging device according to (7), in which

in accordance with a result of the determination by the flatnessdetermining unit, the defect determining unit is configured to determinewhether the pixel of interest is a defective pixel or not, and thedefect correcting unit is configured to correct the pixel value of thepixel of interest, by different methods.

(9) The solid-state imaging device according to (8), in which

the defect correcting unit is configured to correct the pixel value ofthe pixel of interest by replacing the pixel value of the pixel ofinterest with a pixel value of a pixel selected based on the detecteddirection of the texture, when it is determined that the image formed ofthe pixels of the processing unit area is not a flat image.

(10) The solid-state imaging device according to (9), in which

in the case where the pixel array includes the plurality of pixelsincluding a predetermined number of pixels each having one of the sameexposure time and the same exposure sensitivity and being regularlydisposed as an L-shaped group of pixels, and when the detected directionof the texture is a vertical direction, the defect correcting unit isconfigured to generate the pixel value of the pixel selected based onthe direction of the texture by linear interpolation.

(11) The solid-state imaging device according to (10), in which

the defect correcting unit is configured to mix the pixel valuegenerated by the linear interpolation and the pixel value of the pixelof interest, based on a mixing ratio determined based on the pixel valuegenerated by the linear interpolation.

(12) The solid-state imaging device according to (6), further including

a gain adding unit configured to multiply, among the pixels of theprocessing unit area, a pixel value of a pixel having one of a firstexposure time and a first exposure sensitivity by a predetermined gain,to thereby normalize the pixel values of the pixels of the processingunit area, with a pixel value of a pixel having one of a second exposuretime and a second exposure sensitivity being as a reference.

(13) The solid-state imaging device according to any one of (1) to (12),in which

the solid-state imaging device includes a lamination-type image sensorincluding

-   -   a first chip on which the pixel array is disposed, and    -   a second chip including a circuit for achieving a function of        the pixel value correcting unit.

(14) A solid-state imaging method, including

correcting, among pixel values obtained from a plurality of pixels in apixel array, a pixel value of a pixel of the plurality of pixels thatapplies to a preset condition, by using a pixel value of another pixelof the plurality of pixels, the plurality of pixels each having one of adifferent exposure time and a different exposure sensitivity and beingdisposed according to a predetermined rule.

(15) An electronic apparatus, including

a solid-state imaging device including

-   -   a pixel array including a plurality of pixels, the plurality of        pixels each having one of a different exposure time and a        different exposure sensitivity and being disposed according to a        predetermined rule, and    -   a pixel value correcting unit configured to correct, among pixel        values obtained from the plurality of pixels in the pixel array,        a pixel value of a pixel of the plurality of pixels that applies        to a preset condition, by using a pixel value of another pixel        of the plurality of pixels.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. A solid-state imaging device, comprising: a pixelarray including a plurality of pixels, the plurality of pixels eachhaving one of a different exposure time and a different exposuresensitivity and being disposed according to a predetermined rule; and apixel value correcting unit configured to correct, among pixel valuesobtained from the plurality of pixels in the pixel array, a pixel valueof a pixel of the plurality of pixels that applies to a presetcondition, by using a pixel value of another pixel of the plurality ofpixels.
 2. The solid-state imaging device according to claim 1, whereinthe pixel array includes the plurality of pixels each having one of thesame exposure time and the same exposure sensitivity and being regularlydisposed on a row-by-row basis.
 3. The solid-state imaging deviceaccording to claim 1, wherein the pixel array includes the plurality ofpixels including a predetermined number of pixels each having one of thesame exposure time and the same exposure sensitivity and being regularlydisposed as an L-shaped group of pixels.
 4. The solid-state imagingdevice according to claim 1, wherein the pixel value correcting unit isconfigured to set a pixel of interest in the plurality of pixelsdisposed in the pixel array to be the center of the pixel array, extracta processing unit area including a preset number of rows of pixels, andcorrect the pixel value of the pixel of interest for each processingunit area.
 5. The solid-state imaging device according to claim 4,wherein the processing unit area includes five rows.
 6. The solid-stateimaging device according to claim 4, wherein the pixel value correctingunit includes a saturation determining unit configured to determinewhether the processing unit area is saturated or not based on the numberof pixels that output a maximum pixel value among the pixels of theprocessing unit area, a flatness determining unit configured todetermine whether or not an image formed of the pixels of the processingunit area is a flat image that is free from a texture, a directiondetecting unit configured to detect a direction of the texture when itis determined that the image formed of the pixels of the processing unitarea is not a flat image, a defect determining unit configured todetermine whether the pixel of interest is a defective pixel or not, anda defect correcting unit configured to correct the pixel value of thepixel of interest when it is determined that the pixel of interest is adefective pixel.
 7. The solid-state imaging device according to claim 6,wherein in accordance with a result of the determination by thesaturation determining unit, the flatness determining unit is configuredto determine whether the image is a flat image or not, and the directiondetecting unit is configured to detect the direction of the texture, bydifferent methods.
 8. The solid-state imaging device according to claim7, wherein in accordance with a result of the determination by theflatness determining unit, the defect determining unit is configured todetermine whether the pixel of interest is a defective pixel or not, andthe defect correcting unit is configured to correct the pixel value ofthe pixel of interest, by different methods.
 9. The solid-state imagingdevice according to claim 8, wherein the defect correcting unit isconfigured to correct the pixel value of the pixel of interest byreplacing the pixel value of the pixel of interest with a pixel value ofa pixel selected based on the detected direction of the texture, when itis determined that the image formed of the pixels of the processing unitarea is not a flat image.
 10. The solid-state imaging device accordingto claim 9, wherein in the case where the pixel array includes theplurality of pixels including a predetermined number of pixels eachhaving one of the same exposure time and the same exposure sensitivityand being regularly disposed as an L-shaped group of pixels, and whenthe detected direction of the texture is a vertical direction, thedefect correcting unit is configured to generate the pixel value of thepixel selected based on the direction of the texture by linearinterpolation.
 11. The solid-state imaging device according to claim 10,wherein the defect correcting unit is configured to mix the pixel valuegenerated by the linear interpolation and the pixel value of the pixelof interest, based on a mixing ratio determined based on the pixel valuegenerated by the linear interpolation.
 12. The solid-state imagingdevice according to claim 6, further comprising a gain adding unitconfigured to multiply, among the pixels of the processing unit area, apixel value of a pixel having one of a first exposure time and a firstexposure sensitivity by a predetermined gain, to thereby normalize thepixel values of the pixels of the processing unit area, with a pixelvalue of a pixel having one of a second exposure time and a secondexposure sensitivity being as a reference.
 13. The solid-state imagingdevice according to claim 1, wherein the solid-state imaging deviceincludes a lamination-type image sensor including a first chip on whichthe pixel array is disposed, and a second chip including a circuit forachieving a function of the pixel value correcting unit.
 14. Asolid-state imaging method, comprising correcting, among pixel valuesobtained from a plurality of pixels in a pixel array, a pixel value of apixel of the plurality of pixels that applies to a preset condition, byusing a pixel value of another pixel of the plurality of pixels, theplurality of pixels each having one of a different exposure time and adifferent exposure sensitivity and being disposed according to apredetermined rule.
 15. An electronic apparatus, comprising asolid-state imaging device including a pixel array including a pluralityof pixels, the plurality of pixels each having one of a differentexposure time and a different exposure sensitivity and being disposedaccording to a predetermined rule, and a pixel value correcting unitconfigured to correct, among pixel values obtained from the plurality ofpixels in the pixel array, a pixel value of a pixel of the plurality ofpixels that applies to a preset condition, by using a pixel value ofanother pixel of the plurality of pixels.